Knowledge Hub

Getting Started with Kafka on CleanStart

Running Kafka with CleanStart

CleanStart provides production-ready Kafka base images (cleanstart/kafka) with Kafka 3.6 and 3.7 versions that include current security patches. These images support KRaft mode, which eliminates the need for ZooKeeper and significantly simplifies operational complexity. They come with TLS and SASL pre-configured for secure authentication and include partition replication and failover capabilities optimized for Kubernetes deployments.

Quick Start: Single Kafka Broker

Step 1: Generate Cluster ID

Generate a unique cluster ID for your Kafka cluster:

CLUSTER_ID=$(docker run --rm cleanstart/kafka:3.7 \    /opt/kafka/bin/kafka-storage.sh random-uuid)echo "Cluster ID: $CLUSTER_ID"

Step 2: Run Kafka Broker

Start a Kafka broker with KRaft mode:

docker run -d \    --name kafka-broker-1 \    -e CLUSTER_ID=$CLUSTER_ID \    -e KAFKA_NODE_ID=1 \    -e KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://kafka-broker-1:9092" \    -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="PLAINTEXT:PLAINTEXT" \    -e KAFKA_INTER_BROKER_LISTENER_NAME="PLAINTEXT" \    -e KAFKA_CONTROLLER_QUORUM_VOTERS="1@kafka-broker-1:9093" \    -e KAFKA_PROCESS_ROLES="broker,controller" \    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \    -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \    -e KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER" \    -e KAFKA_LISTENERS="PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" \    -v kafka-data:/var/lib/kafka/data \    -p 9092:9092 \    cleanstart/kafka:3.7

Step 3: Create a Topic

Create a topic for message publishing:

docker exec kafka-broker-1 \    /opt/kafka/bin/kafka-topics.sh \    --bootstrap-server localhost:9092 \    --create \    --topic my-topic \    --partitions 3 \    --replication-factor 1

Step 4: Produce and Consume Messages

Test message streaming by producing and consuming messages:

# Terminal 1: Start consumerdocker exec -it kafka-broker-1 \    /opt/kafka/bin/kafka-console-consumer.sh \    --bootstrap-server localhost:9092 \    --topic my-topic \    --from-beginning # Terminal 2: Send messagesdocker exec -it kafka-broker-1 \    /opt/kafka/bin/kafka-console-producer.sh \    --bootstrap-server localhost:9092 \    --topic my-topic # Type messages in Terminal 2, see them in Terminal 1

Multi-Broker Cluster Setup

Docker Compose: 3-Broker Cluster

Set up a complete Kafka cluster with three brokers:

version: '3.9' services:  # Generate cluster ID once  kafka-generate-uuid:    image: cleanstart/kafka:3.7    entrypoint: sh -c "echo $$(./bin/kafka-storage.sh random-uuid) > /cluster-id.txt"    volumes:      - cluster-id:/cluster-id   kafka-broker-1:    image: cleanstart/kafka:3.7    container_name: kafka-broker-1    depends_on:      kafka-generate-uuid:        condition: service_completed_successfully    environment:      CLUSTER_ID: ${CLUSTER_ID:-abc123}      KAFKA_NODE_ID: 1      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-broker-1:9092,CONTROLLER://kafka-broker-1:9093"      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093"      KAFKA_PROCESS_ROLES: "broker,controller"      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093"      KAFKA_LOG_RETENTION_HOURS: 168    ports:      - "9092:9092"    volumes:      - kafka-data-1:/var/lib/kafka/data    networks:      - kafka-network   kafka-broker-2:    image: cleanstart/kafka:3.7    container_name: kafka-broker-2    depends_on:      kafka-generate-uuid:        condition: service_completed_successfully    environment:      CLUSTER_ID: ${CLUSTER_ID:-abc123}      KAFKA_NODE_ID: 2      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-broker-2:9092,CONTROLLER://kafka-broker-2:9093"      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093"      KAFKA_PROCESS_ROLES: "broker,controller"      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093"    volumes:      - kafka-data-2:/var/lib/kafka/data    networks:      - kafka-network   kafka-broker-3:    image: cleanstart/kafka:3.7    container_name: kafka-broker-3    depends_on:      kafka-generate-uuid:        condition: service_completed_successfully    environment:      CLUSTER_ID: ${CLUSTER_ID:-abc123}      KAFKA_NODE_ID: 3      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka-broker-3:9092,CONTROLLER://kafka-broker-3:9093"      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093"      KAFKA_PROCESS_ROLES: "broker,controller"      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093"    volumes:      - kafka-data-3:/var/lib/kafka/data    networks:      - kafka-network volumes:  kafka-data-1:  kafka-data-2:  kafka-data-3:  cluster-id: networks:  kafka-network:    driver: bridge

Deploy:

docker-compose up -d

Kafka Configuration

Essential Broker Settings

Essential broker configuration controls replication, durability, and performance characteristics of your Kafka cluster. For replication and durability, the min.insync.replicas parameter specifies the minimum number of in-sync replicas required before a producer can receive an ACK (typically set to 2 for safety), the default.replication.factor determines how many copies of each partition exist across the cluster (typically 3 for high availability), and offsets.topic.replication.factor controls how the consumer offset tracking topic is replicated. For retention policies, log.retention.hours defines how long messages are kept in the log (commonly 7 days or 168 hours), and log.retention.bytes sets a size-based limit per partition (typically 1GB). For performance tuning, num.network.threads controls the number of network I/O threads (commonly 8), num.io.threads handles disk I/O operations (typically 8), and socket buffer sizes like socket.send.buffer.bytes and socket.receive.buffer.bytes optimize network communication (typically 100KB). For compression and cleanup, compression.type can be set to snappy for default compression, and log.cleanup.policy defines whether old logs are deleted or compacted.

# Replication and durabilitymin.insync.replicas=2                    # Minimum replicas for ACKdefault.replication.factor=3              # Default replicationoffsets.topic.replication.factor=3        # Offsets topic replication # Retention policylog.retention.hours=168                   # Keep logs 7 dayslog.retention.bytes=1073741824            # Keep 1GB per partition # Performancenum.network.threads=8                     # Network I/O threadsnum.io.threads=8                          # Disk I/O threadssocket.send.buffer.bytes=102400           # Socket send buffersocket.receive.buffer.bytes=102400        # Socket receive buffer # Compressioncompression.type=snappy                   # Default compressionlog.cleanup.policy=delete                 # Cleanup policy

Enable in Docker:

docker run -d \    --name kafka-broker \    -e KAFKA_MIN_INSYNC_REPLICAS=2 \    -e KAFKA_DEFAULT_REPLICATION_FACTOR=3 \    -e KAFKA_LOG_RETENTION_HOURS=168 \    -e KAFKA_COMPRESSION_TYPE=snappy \    cleanstart/kafka:3.7

TLS/SASL Authentication

Generate Certificates

# Create truststore and keystorekeytool -genkey -alias kafka-broker -keystore kafka.server.keystore.jks \    -storepass changeit -keypass changeit -dname "CN=kafka-broker" keytool -export -alias kafka-broker -keystore kafka.server.keystore.jks \    -storepass changeit -file kafka-broker.crt keytool -import -alias kafka-broker -file kafka-broker.crt \    -keystore kafka.server.truststore.jks -storepass changeit -noprompt

Enable in Broker

FROM cleanstart/kafka:3.7 COPY kafka.server.keystore.jks /etc/kafka/secrets/COPY kafka.server.truststore.jks /etc/kafka/secrets/ ENV KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SSLENV KAFKA_SSL_KEYSTORE_LOCATION=/etc/kafka/secrets/kafka.server.keystore.jksENV KAFKA_SSL_KEYSTORE_PASSWORD=changeitENV KAFKA_SSL_KEY_PASSWORD=changeitENV KAFKA_SSL_TRUSTSTORE_LOCATION=/etc/kafka/secrets/kafka.server.truststore.jksENV KAFKA_SSL_TRUSTSTORE_PASSWORD=changeit

Topic Management

Create Topics with Retention

# Create topic with 7-day retentiondocker exec kafka-broker-1 \    /opt/kafka/bin/kafka-topics.sh \    --bootstrap-server localhost:9092 \    --create \    --topic events \    --partitions 6 \    --replication-factor 3 \    --config retention.ms=604800000 # List topicsdocker exec kafka-broker-1 \    /opt/kafka/bin/kafka-topics.sh \    --bootstrap-server localhost:9092 \    --list # Describe topicdocker exec kafka-broker-1 \    /opt/kafka/bin/kafka-topics.sh \    --bootstrap-server localhost:9092 \    --describe \    --topic events

Modify Topic Configuration

docker exec kafka-broker-1 \    /opt/kafka/bin/kafka-configs.sh \    --bootstrap-server localhost:9092 \    --entity-type topics \    --entity-name events \    --alter \    --add-config retention.ms=2592000000

Kubernetes Deployment

Create kafka-statefulset.yaml:

apiVersion: apps/v1kind: StatefulSetmetadata:  name: kafkaspec:  serviceName: kafka  replicas: 3  selector:    matchLabels:      app: kafka  template:    metadata:      labels:        app: kafka    spec:      containers:      - name: kafka        image: cleanstart/kafka:3.7        ports:        - containerPort: 9092          name: client        - containerPort: 9093          name: controller        env:        - name: CLUSTER_ID          value: "abc123def456"        - name: KAFKA_NODE_ID          valueFrom:            fieldRef:              fieldPath: metadata.name        - name: KAFKA_ADVERTISED_LISTENERS          value: "PLAINTEXT://$(POD_NAME).kafka-headless:9092"        - name: KAFKA_PROCESS_ROLES          value: "broker,controller"        - name: KAFKA_CONTROLLER_QUORUM_VOTERS          value: "0@kafka-0.kafka-headless:9093,1@kafka-1.kafka-headless:9093,2@kafka-2.kafka-headless:9093"        volumeMounts:        - name: data          mountPath: /var/lib/kafka/data        livenessProbe:          exec:            command:            - /opt/kafka/bin/kafka-broker-api-versions.sh            - --bootstrap-server=localhost:9092          initialDelaySeconds: 30          periodSeconds: 10   volumeClaimTemplates:  - metadata:      name: data    spec:      accessModes: ["ReadWriteOnce"]      resources:        requests:          storage: 10Gi ---apiVersion: v1kind: Servicemetadata:  name: kafka-headlessspec:  clusterIP: None  ports:  - port: 9092    name: client  - port: 9093    name: controller  selector:    app: kafka

Deploy:

kubectl apply -f kafka-statefulset.yaml

Monitoring and Health

Check Cluster Status

docker exec kafka-broker-1 \    /opt/kafka/bin/kafka-metadata.sh \    --snapshot /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log \    --print

Monitor Broker Metrics

docker exec kafka-broker-1 \    /opt/kafka/bin/kafka-run-class.sh \    kafka.tools.JmxTool

Consumer Lag

docker exec kafka-broker-1 \    /opt/kafka/bin/kafka-consumer-groups.sh \    --bootstrap-server localhost:9092 \    --group my-consumer-group \    --describe

Troubleshooting

Brokers that fail to start typically have mismatched CLUSTER_ID and KAFKA_NODE_ID values—verify that these identifiers correspond correctly in your configuration across all brokers. Topics that won't replicate usually indicate insufficient broker availability or incorrect replication factor settings—ensure that enough brokers are running and accessible to satisfy your replication factor requirements. Growing consumer lag suggests that consumer throughput is not keeping pace with producer throughput or that broker performance is degraded—monitor both the producer and consumer rates and check broker CPU and disk I/O metrics. Leader election issues stem from having fewer than a majority of controllers running in the cluster—maintain at least three controller nodes to ensure quorum and enable stable leadership elections.

Image Options

Image	Use Case
`cleanstart/kafka:3.7`	Stable LTS (recommended)
`cleanstart/kafka:3.6`	Legacy support

CleanStart Production Hardening

Read-Only Root Filesystem

Enforce immutable infrastructure with read-only root, with writable paths for Kafka logs and data:

apiVersion: v1kind: Podmetadata:  name: kafka-brokerspec:  containers:  - name: kafka    image: cleanstart/kafka:3.7    securityContext:      readOnlyRootFilesystem: true    volumeMounts:    - name: data      mountPath: /var/lib/kafka/data    - name: logs      mountPath: /var/log/kafka    - name: tmp      mountPath: /tmp  volumes:  - name: data    persistentVolumeClaim:      claimName: kafka-data  - name: logs    emptyDir: {}  - name: tmp    emptyDir: {}

Shell-Less ENTRYPOINT

Remove shell for attack surface reduction. Update your Dockerfile:

FROM cleanstart/kafka:3.7 WORKDIR /opt/kafka # Copy custom Kafka configurationCOPY server.properties config/server.properties # Declarative Image Builder: Use cleanimg-init as PID 1ENTRYPOINT ["/cleanimg-init", "--"]CMD ["bin/kafka-server-start.sh", "config/server.properties"]

cleanimg-customize: Kafka Configuration

Inject production security and performance settings:

FROM cleanstart/kafka:3.7 WORKDIR /opt/kafka # Create custom configuration with security hardeningRUN echo 'security.protocol=PLAINTEXT' >> config/server.properties && \    echo 'auto.create.topics.enable=false' >> config/server.properties && \    echo 'log.retention.hours=168' >> config/server.properties && \    echo 'num.replica.fetchers=2' >> config/server.properties ENTRYPOINT ["/cleanimg-init", "--"]CMD ["bin/kafka-server-start.sh", "config/server.properties"]

Security Context

Complete Kubernetes securityContext for hardened Kafka containers:

apiVersion: apps/v1kind: StatefulSetmetadata:  name: kafkaspec:  template:    spec:      securityContext:        fsGroup: 1000        seccompProfile:          type: RuntimeDefault      containers:      - name: kafka        image: cleanstart/kafka:3.7        securityContext:          allowPrivilegeEscalation: false          readOnlyRootFilesystem: true          runAsNonRoot: true          runAsUser: 1000          runAsGroup: 1000          capabilities:            drop:            - ALL            add:            - NET_BIND_SERVICE        resources:          requests:            cpu: 1000m            memory: 1Gi          limits:            cpu: 2000m            memory: 2Gi

Next Steps

Advanced Kafka Configuration, Kafka Monitoring and Alerting, and Kafka Connect Integration.