Running Kafka with CleanStart
CleanStart provides production-ready Kafka base images (cleanstart/kafka) with Kafka 3.6 and 3.7 versions that include current security patches. These images support KRaft mode, which eliminates the need for ZooKeeper and significantly simplifies operational complexity. They come with TLS and SASL pre-configured for secure authentication and include partition replication and failover capabilities optimized for Kubernetes deployments.
Quick Start: Single Kafka Broker
Step 1: Generate Cluster ID
Generate a unique cluster ID for your Kafka cluster:
CLUSTER_ID=$(docker run --rm cleanstart/kafka:3.7 \ /opt/kafka/bin/kafka-storage.sh random-uuid)echo "Cluster ID: $CLUSTER_ID"Step 2: Run Kafka Broker
Start a Kafka broker with KRaft mode:
docker run -d \ --name kafka-broker-1 \ -e CLUSTER_ID=$CLUSTER_ID \ -e KAFKA_NODE_ID=1 \ -e KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://kafka-broker-1:9092" \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="PLAINTEXT:PLAINTEXT" \ -e KAFKA_INTER_BROKER_LISTENER_NAME="PLAINTEXT" \ -e KAFKA_CONTROLLER_QUORUM_VOTERS="1@kafka-broker-1:9093" \ -e KAFKA_PROCESS_ROLES="broker,controller" \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \ -e KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER" \ -e KAFKA_LISTENERS="PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" \ -v kafka-data:/var/lib/kafka/data \ -p 9092:9092 \ cleanstart/kafka:3.7Step 3: Create a Topic
Create a topic for message publishing:
docker exec kafka-broker-1 \ /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --create \ --topic my-topic \ --partitions 3 \ --replication-factor 1Step 4: Produce and Consume Messages
Test message streaming by producing and consuming messages:
# Terminal 1: Start consumerdocker exec -it kafka-broker-1 \ /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9092 \ --topic my-topic \ --from-beginning # Terminal 2: Send messagesdocker exec -it kafka-broker-1 \ /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic my-topic # Type messages in Terminal 2, see them in Terminal 1Multi-Broker Cluster Setup
Docker Compose: 3-Broker Cluster
Set up a complete Kafka cluster with three brokers:
version: '3.9' services: # Generate cluster ID once kafka-generate-uuid: image: cleanstart/kafka:3.7 entrypoint: sh -c "echo $$(./bin/kafka-storage.sh random-uuid) > /cluster-id.txt" volumes: - cluster-id:/cluster-id kafka-broker-1: image: cleanstart/kafka:3.7 container_name: kafka-broker-1 depends_on: kafka-generate-uuid: condition: service_completed_successfully environment: CLUSTER_ID: ${CLUSTER_ID:-abc123} KAFKA_NODE_ID: 1 KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-broker-1:9092,CONTROLLER://kafka-broker-1:9093" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093" KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" KAFKA_LOG_RETENTION_HOURS: 168 ports: - "9092:9092" volumes: - kafka-data-1:/var/lib/kafka/data networks: - kafka-network kafka-broker-2: image: cleanstart/kafka:3.7 container_name: kafka-broker-2 depends_on: kafka-generate-uuid: condition: service_completed_successfully environment: CLUSTER_ID: ${CLUSTER_ID:-abc123} KAFKA_NODE_ID: 2 KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-broker-2:9092,CONTROLLER://kafka-broker-2:9093" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093" KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" volumes: - kafka-data-2:/var/lib/kafka/data networks: - kafka-network kafka-broker-3: image: cleanstart/kafka:3.7 container_name: kafka-broker-3 depends_on: kafka-generate-uuid: condition: service_completed_successfully environment: CLUSTER_ID: ${CLUSTER_ID:-abc123} KAFKA_NODE_ID: 3 KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-broker-3:9092,CONTROLLER://kafka-broker-3:9093" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093" KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" volumes: - kafka-data-3:/var/lib/kafka/data networks: - kafka-network volumes: kafka-data-1: kafka-data-2: kafka-data-3: cluster-id: networks: kafka-network: driver: bridgeDeploy:
docker-compose up -dKafka Configuration
Essential Broker Settings
Essential broker configuration controls replication, durability, and performance characteristics of your Kafka cluster. For replication and durability, the min.insync.replicas parameter specifies the minimum number of in-sync replicas required before a producer can receive an ACK (typically set to 2 for safety), the default.replication.factor determines how many copies of each partition exist across the cluster (typically 3 for high availability), and offsets.topic.replication.factor controls how the consumer offset tracking topic is replicated. For retention policies, log.retention.hours defines how long messages are kept in the log (commonly 7 days or 168 hours), and log.retention.bytes sets a size-based limit per partition (typically 1GB). For performance tuning, num.network.threads controls the number of network I/O threads (commonly 8), num.io.threads handles disk I/O operations (typically 8), and socket buffer sizes like socket.send.buffer.bytes and socket.receive.buffer.bytes optimize network communication (typically 100KB). For compression and cleanup, compression.type can be set to snappy for default compression, and log.cleanup.policy defines whether old logs are deleted or compacted.
# Replication and durabilitymin.insync.replicas=2 # Minimum replicas for ACKdefault.replication.factor=3 # Default replicationoffsets.topic.replication.factor=3 # Offsets topic replication # Retention policylog.retention.hours=168 # Keep logs 7 dayslog.retention.bytes=1073741824 # Keep 1GB per partition # Performancenum.network.threads=8 # Network I/O threadsnum.io.threads=8 # Disk I/O threadssocket.send.buffer.bytes=102400 # Socket send buffersocket.receive.buffer.bytes=102400 # Socket receive buffer # Compressioncompression.type=snappy # Default compressionlog.cleanup.policy=delete # Cleanup policyEnable in Docker:
docker run -d \ --name kafka-broker \ -e KAFKA_MIN_INSYNC_REPLICAS=2 \ -e KAFKA_DEFAULT_REPLICATION_FACTOR=3 \ -e KAFKA_LOG_RETENTION_HOURS=168 \ -e KAFKA_COMPRESSION_TYPE=snappy \ cleanstart/kafka:3.7TLS/SASL Authentication
Generate Certificates
# Create truststore and keystorekeytool -genkey -alias kafka-broker -keystore kafka.server.keystore.jks \ -storepass changeit -keypass changeit -dname "CN=kafka-broker" keytool -export -alias kafka-broker -keystore kafka.server.keystore.jks \ -storepass changeit -file kafka-broker.crt keytool -import -alias kafka-broker -file kafka-broker.crt \ -keystore kafka.server.truststore.jks -storepass changeit -nopromptEnable in Broker
FROM cleanstart/kafka:3.7 COPY kafka.server.keystore.jks /etc/kafka/secrets/COPY kafka.server.truststore.jks /etc/kafka/secrets/ ENV KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SSLENV KAFKA_SSL_KEYSTORE_LOCATION=/etc/kafka/secrets/kafka.server.keystore.jksENV KAFKA_SSL_KEYSTORE_PASSWORD=changeitENV KAFKA_SSL_KEY_PASSWORD=changeitENV KAFKA_SSL_TRUSTSTORE_LOCATION=/etc/kafka/secrets/kafka.server.truststore.jksENV KAFKA_SSL_TRUSTSTORE_PASSWORD=changeitTopic Management
Create Topics with Retention
# Create topic with 7-day retentiondocker exec kafka-broker-1 \ /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --create \ --topic events \ --partitions 6 \ --replication-factor 3 \ --config retention.ms=604800000 # List topicsdocker exec kafka-broker-1 \ /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --list # Describe topicdocker exec kafka-broker-1 \ /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --describe \ --topic eventsModify Topic Configuration
docker exec kafka-broker-1 \ /opt/kafka/bin/kafka-configs.sh \ --bootstrap-server localhost:9092 \ --entity-type topics \ --entity-name events \ --alter \ --add-config retention.ms=2592000000Kubernetes Deployment
Create kafka-statefulset.yaml:
apiVersion: apps/v1kind: StatefulSetmetadata: name: kafkaspec: serviceName: kafka replicas: 3 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: containers: - name: kafka image: cleanstart/kafka:3.7 ports: - containerPort: 9092 name: client - containerPort: 9093 name: controller env: - name: CLUSTER_ID value: "abc123def456" - name: KAFKA_NODE_ID valueFrom: fieldRef: fieldPath: metadata.name - name: KAFKA_ADVERTISED_LISTENERS value: "PLAINTEXT://$(POD_NAME).kafka-headless:9092" - name: KAFKA_PROCESS_ROLES value: "broker,controller" - name: KAFKA_CONTROLLER_QUORUM_VOTERS value: "0@kafka-0.kafka-headless:9093,1@kafka-1.kafka-headless:9093,2@kafka-2.kafka-headless:9093" volumeMounts: - name: data mountPath: /var/lib/kafka/data livenessProbe: exec: command: - /opt/kafka/bin/kafka-broker-api-versions.sh - --bootstrap-server=localhost:9092 initialDelaySeconds: 30 periodSeconds: 10 volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi ---apiVersion: v1kind: Servicemetadata: name: kafka-headlessspec: clusterIP: None ports: - port: 9092 name: client - port: 9093 name: controller selector: app: kafkaDeploy:
kubectl apply -f kafka-statefulset.yamlMonitoring and Health
Check Cluster Status
docker exec kafka-broker-1 \ /opt/kafka/bin/kafka-metadata.sh \ --snapshot /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log \ --printMonitor Broker Metrics
docker exec kafka-broker-1 \ /opt/kafka/bin/kafka-run-class.sh \ kafka.tools.JmxToolConsumer Lag
docker exec kafka-broker-1 \ /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --group my-consumer-group \ --describeTroubleshooting
Brokers that fail to start typically have mismatched CLUSTER_ID and KAFKA_NODE_ID values—verify that these identifiers correspond correctly in your configuration across all brokers. Topics that won't replicate usually indicate insufficient broker availability or incorrect replication factor settings—ensure that enough brokers are running and accessible to satisfy your replication factor requirements. Growing consumer lag suggests that consumer throughput is not keeping pace with producer throughput or that broker performance is degraded—monitor both the producer and consumer rates and check broker CPU and disk I/O metrics. Leader election issues stem from having fewer than a majority of controllers running in the cluster—maintain at least three controller nodes to ensure quorum and enable stable leadership elections.
Image Options
Image | Use Case |
|---|---|
| Stable LTS (recommended) |
| Legacy support |
CleanStart Production Hardening
Read-Only Root Filesystem
Enforce immutable infrastructure with read-only root, with writable paths for Kafka logs and data:
apiVersion: v1kind: Podmetadata: name: kafka-brokerspec: containers: - name: kafka image: cleanstart/kafka:3.7 securityContext: readOnlyRootFilesystem: true volumeMounts: - name: data mountPath: /var/lib/kafka/data - name: logs mountPath: /var/log/kafka - name: tmp mountPath: /tmp volumes: - name: data persistentVolumeClaim: claimName: kafka-data - name: logs emptyDir: {} - name: tmp emptyDir: {}Shell-Less ENTRYPOINT
Remove shell for attack surface reduction. Update your Dockerfile:
FROM cleanstart/kafka:3.7 WORKDIR /opt/kafka # Copy custom Kafka configurationCOPY server.properties config/server.properties # Declarative Image Builder: Use cleanimg-init as PID 1ENTRYPOINT ["/cleanimg-init", "--"]CMD ["bin/kafka-server-start.sh", "config/server.properties"]cleanimg-customize: Kafka Configuration
Inject production security and performance settings:
FROM cleanstart/kafka:3.7 WORKDIR /opt/kafka # Create custom configuration with security hardeningRUN echo 'security.protocol=PLAINTEXT' >> config/server.properties && \ echo 'auto.create.topics.enable=false' >> config/server.properties && \ echo 'log.retention.hours=168' >> config/server.properties && \ echo 'num.replica.fetchers=2' >> config/server.properties ENTRYPOINT ["/cleanimg-init", "--"]CMD ["bin/kafka-server-start.sh", "config/server.properties"]Security Context
Complete Kubernetes securityContext for hardened Kafka containers:
apiVersion: apps/v1kind: StatefulSetmetadata: name: kafkaspec: template: spec: securityContext: fsGroup: 1000 seccompProfile: type: RuntimeDefault containers: - name: kafka image: cleanstart/kafka:3.7 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 runAsGroup: 1000 capabilities: drop: - ALL add: - NET_BIND_SERVICE resources: requests: cpu: 1000m memory: 1Gi limits: cpu: 2000m memory: 2GiNext Steps
Advanced Kafka Configuration, Kafka Monitoring and Alerting, and Kafka Connect Integration.