Install Apache Kafka on VPS - Event Streaming Platform

June 2026 · 9 min read

Apache Kafka is a distributed message broker designed for event streaming — capable of sending millions of messages per second with high throughput, making it ideal for real-time data pipelines. Unlike RabbitMQ which excels at point-to-point messaging, Kafka is built for fan-out scenarios where multiple consumers read the same events and data can be replayed for extended periods. This guide walks through setting up a Kafka cluster using Docker Compose, complete with a Kafka UI dashboard, producer/consumer examples, and topic management.

Apache Kafka Architecture Overview

Broker: Stores messages and serves clients — production setups require 3+ brokers for high availability
Topic: Named channel where producers send messages and consumers read from (similar to a queue name)
Partition: Topics are divided into partitions for parallelization — each consumer reads from one partition
Consumer Group: Set of consumers reading the same topic — enables load balancing and offset management
Replication Factor: Number of copies each partition maintains across brokers — ensures fault tolerance if a broker fails
KRaft Mode: Modern Kafka (v3.3+) uses KRaft instead of Zookeeper — metadata stored in brokers directly

Install Kafka with Docker Compose

mkdir -p /opt/kafka && cd /opt/kafka

# Create docker-compose.yml
cat > docker-compose.yml << 'EOF'
version: "3.8"

services:
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    restart: unless-stopped
    container_name: kafka-broker
    ports:
      - "127.0.0.1:9092:9092"
      - "127.0.0.1:29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      CLUSTER_ID: MkwNYEubTXePEKdc6tH9kQ

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    restart: unless-stopped
    container_name: kafka-ui
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: kafka-cluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092

volumes:
  kafka_data:
EOF

docker-compose up -d

Set Up Nginx Reverse Proxy for Kafka UI

nano /etc/nginx/sites-available/kafka-ui

server {
    listen 80;
    server_name kafka.yourdomain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name kafka.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/kafka.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/kafka.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}

ln -s /etc/nginx/sites-available/kafka-ui /etc/nginx/sites-enabled/
nginx -t && systemctl reload nginx
certbot --nginx -d kafka.yourdomain.com

Create Topics and Producer/Consumer

# Verify Kafka is ready
docker exec kafka-broker kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# Create a topic named "orders" with 3 partitions
docker exec kafka-broker kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders \
  --partitions 3 \
  --replication-factor 1

# List all topics
docker exec kafka-broker kafka-topics.sh --bootstrap-server localhost:9092 --list

# Describe topic details
docker exec kafka-broker kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

Producer - Send Messages to Topic

# Start a console producer for the "orders" topic
docker exec -it kafka-broker kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders

# Type messages and press Enter (message is sent to topic)
# Example messages:
{"order_id": 1001, "customer": "Alice", "amount": 500}
{"order_id": 1002, "customer": "Bob", "amount": 1200}
{"order_id": 1003, "customer": "Charlie", "amount": 750}

Consumer - Read Messages from Topic

# Start a console consumer for the "orders" topic (read all messages from beginning)
docker exec -it kafka-broker kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --from-beginning \
  --group my-consumer-group

# To read only new messages (from now onwards)
docker exec -it kafka-broker kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --group my-consumer-group

# List all consumer groups
docker exec kafka-broker kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --list

# Describe consumer group offsets
docker exec kafka-broker kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group my-consumer-group

Use Kafka Producer/Consumer in Application (Python)

# Install library
pip install kafka-python

# Producer (send_order.py)
from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Send 10 messages
for i in range(10):
    message = {
        'order_id': 1000 + i,
        'customer': f'Customer{i}',
        'amount': 100 * (i + 1)
    }
    producer.send('orders', value=message)
    print(f"Sent: {message}")

producer.close()

# Consumer (consume_order.py)
from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers=['localhost:9092'],
    group_id='python-consumer',
    auto_offset_reset='earliest',
    value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)

for message in consumer:
    print(f"Received: {message.value}")
    print(f"Partition: {message.partition}, Offset: {message.offset}")

Install Kafka Connect for Data Pipeline

# Add to docker-compose.yml (before volumes:)
  kafka-connect:
    image: confluentinc/cp-kafka-connect:7.5.0
    restart: unless-stopped
    container_name: kafka-connect
    ports:
      - "127.0.0.1:8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_GROUP_ID: connect-cluster
      CONNECT_CONFIG_STORAGE_TOPIC: _connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: _connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: _connect-status
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
      CONNECT_REST_PORT: 8083
    depends_on:
      - kafka

# Restart docker-compose
docker-compose down && docker-compose up -d

# Verify Kafka Connect is ready
curl -s http://localhost:8083 | python3 -m json.tool

# List available plugins
curl -s http://localhost:8083/connector-plugins | python3 -m json.tool

Tip: In production, set replication factor to 3 or higher and run at least 3 brokers to ensure cluster survives a broker failure. Additionally, configure retention.ms to automatically delete old messages and manage disk space efficiently.

Verify Kafka is Working Correctly

# Check container status
docker-compose ps

# Check Kafka broker status via Kafka UI API
curl -s http://localhost:8080/api/clusters/kafka-cluster | python3 -m json.tool

# List all topics
docker exec kafka-broker kafka-topics.sh --bootstrap-server localhost:9092 --list

# Check topic log size
docker exec kafka-broker kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --topic-list orders

# Check consumer group offsets
docker exec kafka-broker kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group my-consumer-group

# View broker metrics (JMX)
curl -s http://localhost:9999 2>/dev/null || echo "JMX metrics not enabled"

Tip: Use Kafka UI dashboard (https://kafka.yourdomain.com) to view topics, consumer groups, messages, and broker status visually. This is more convenient than CLI for learning and debugging — use the UI first to understand the system, then switch to CLI commands for production monitoring.

Troubleshooting Common Issues

Error: "Connection refused" when producer/consumer connects

Broker is not yet ready or advertised listener configuration is incorrect — ensure KAFKA_ADVERTISED_LISTENERS points to a host/port that clients can reach.

# Check logs
docker-compose logs kafka

# If broker is still starting, wait 10-15 seconds then restart
docker-compose restart kafka

# Check if port 9092 is listening
netstat -tlnp | grep 9092 || ss -tlnp | grep 9092

Error: "Topic already exists" when creating a topic with same name

Topic has already been created — delete the old one first if you need to recreate it.

# Delete topic
docker exec kafka-broker kafka-topics.sh --bootstrap-server localhost:9092 \
  --delete --topic orders

# Wait 2-3 seconds then create new one
docker exec kafka-broker kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

Consumer not reading messages even after producer sent them

Consumer group offset is pointing ahead — if using --from-beginning, check that auto.offset.reset is set to earliest.

# Check current offset
docker exec kafka-broker kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group my-consumer-group

# Reset offset to earliest
docker exec kafka-broker kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --reset-offsets \
  --to-earliest \
  --execute \
  --group my-consumer-group \
  --topic orders

# Try consumer again
docker exec -it kafka-broker kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --group my-consumer-group

Kafka UI not displaying messages

Usually caused by slow UI startup or broker connection drop — try refreshing the page.

# Check Kafka UI logs
docker-compose logs kafka-ui

# Verify connection from UI to broker
docker exec kafka-ui curl -s http://kafka:29092 -I

# Restart UI if needed
docker-compose restart kafka-ui

Disk space full due to Kafka logs

Topic has unlimited retention policy — set retention.ms to delete old messages automatically.

# Check segment storage
ls -lh /var/lib/docker/volumes/kafka_data/_data/

# Set retention to 24 hours for old messages
docker exec kafka-broker kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --add-config retention.ms=86400000

# Trigger cleanup of old segments
docker exec kafka-broker kafka-log-cleaner.sh --bootstrap-server localhost:9092

Frequently Asked Questions

How is Kafka different from RabbitMQ?

Kafka is designed for event streaming where multiple consumers read the same data (fan-out pattern) and messages are retained for extended periods enabling replay scenarios. RabbitMQ is optimized for point-to-point messaging where each message goes to one consumer and is deleted after consumption. Kafka dominates modern data pipelines requiring data replay and multiple subscribers, while RabbitMQ excels at traditional task queuing.

How many partitions should I create?

A good starting rule is to set partitions equal to the maximum number of consumers you expect in a group, since each consumer reads exactly one partition. If partitions exceed consumers, some partitions stay idle. If consumers exceed partitions, some consumers remain idle. Start with 3 partitions and adjust based on actual throughput and consumer count.

How long can I keep Kafka messages?

Retention depends on retention policy and disk space — configure retention.ms (e.g., 7 days = 604800000 milliseconds) or retention.bytes (e.g., 10GB). Kafka automatically deletes old segments. Unlike RabbitMQ which deletes on consumption, Kafka retains messages so new consumers can replay them.