Deploying Kafka on Docker and Playing with it

Introduction

Hey there! Welcome to this tutorial where we will cover the deployment of Kafka on Docker and how to interact with it by creating producers and consumers. We will be using Python, YAML, Kafka, Docker, and more to accomplish this. Let's dive in!

Youtube Video
 

Setting up Kafka using Docker Compose

To start off, we need to define our Kafka configuration in a Docker Compose file. Here's an example of what it might look like:

```yaml
version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    networks:
      - kafka-network
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    environment:
      KAFKA_ADVERTISED_HOST_NAME: <your_laptop_ip_address>
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LOG_RETENTION_HOURS: 1
      KAFKA_LOG_SEGMENT_BYTES: 1073741824
      KAFKA_RETENTION_CHECK_INTERVAL_MS: 300000
    depends_on:
      - zookeeper
    networks:
      - kafka-network
    ports:
      - "9092:9092"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: "no"
networks:
  kafka-network:
    driver: bridge
```

In this configuration, we have a ZooKeeper container and a Kafka container. We specify the image versions we want to use, along with the necessary environment variables such as the Kafka advertised hostname, ZooKeeper connection, and more.

To start the Kafka containers, navigate to the directory containing the Docker Compose file and run the following command:

docker-compose up

Interacting with Kafka using Python

Now that our Kafka containers are up and running, let's explore how we can interact with Kafka using Python.

First, we need to install necessary Python libraries. You can do this by running the following command:

pip install kafka-python

Next, we will create a Kafka consumer and producer using the `kafka-python` library. Here's an example of how to create a Kafka consumer:

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'test-topic',
    bootstrap_servers='<your_laptop_ip_address>'
)

for message in consumer:
    print(message.value.decode('utf-8'))

In this code snippet, we create a KafkaConsumer object and subscribe to the 'test-topic'. We then loop through the messages received and print out their values.

Next, let's create a Kafka producer:

from kafka import KafkaProducer
import json
import datetime
import random

producer = KafkaProducer(
    bootstrap_servers='<your_laptop_ip_address>'
)

def generate_random_data():
    data = {
        'timestamp': str(datetime.datetime.now()),
        'value_status': 'high' if random.random() > 0.5 else 'low'
    }
    return json.dumps(data).encode('utf-8')

producer.send('test-topic', generate_random_data())
producer.flush()

In this code snippet, we generate random data in JSON format and send it to the 'test-topic' using the Kafka producer.

Running the Kafka Consumer and Producer

To run the Kafka consumer, execute the following Python script:

python consumer.py

You should now see the messages being printed in your terminal as they are consumed from the Kafka topic.

To run the Kafka producer, execute the following Python script:

python producer.py

This will send random data to the Kafka topic.

Congratulations! You have successfully deployed Kafka on Docker and interacted with it using Python. You can find the complete code example and Docker configuration in this https://github.com/ktechhub/deploy_kafka_docker.

Thank you for following along with this tutorial. Feel free to reach out if you have any questions. See you in the next video tutorial, bye!

Links:

GitHub: https://github.com/ktechhub/deploy_kafka_docker
YouTube: https://www.youtube.com/watch?v=CEZ1VJW1duo

Mumuni Mohammed

Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.