Debugging CleanStart Containers

Last updated: April 2026

Overview

CleanStart production images are distroless -- no shell, no debug tools, no package manager. This is intentional: fewer binaries means a smaller attack surface. When you need to troubleshoot, CleanStart provides debug variant images that you attach to running pods via kubectl debug, without modifying or redeploying the production container.

How Debug Images Work

Each CleanStart image that declares debug_packages in its template automatically gets a -debug variant built alongside the production image. The debug variant contains the same application binaries plus a shell and diagnostic tools.

The debug image is never deployed as the running container. Instead, you inject it as an ephemeral container that shares the production pod's process namespace.

Quick Start

Attach a Debug Container

kubectl debug -it my-redis-pod \  --image=gcr.io/clean-image-build/redis:8.2.2-debug \  --target=redis

This creates an ephemeral container inside the running pod. You get a shell with full visibility into the production container's processes.

Inspect the Production Process

# List all processes (production + debug containers share PID namespace)ps -ef # Trace system calls of the production process (PID 1)strace -p 1 -e trace=open,read,write # Check open fileslsof -p 1 # Network connectionsss -tlnp

Exit When Done

exit

The ephemeral container terminates. The production container is unaffected.

Why This Works

CleanStart production pods are generated with shareProcessNamespace: true in the pod spec. This Kubernetes setting allows all containers in the pod (including ephemeral containers added later) to see each other's processes. The production container doesn't need any modification -- the debug container simply joins its namespace.

spec:  shareProcessNamespace: true    # Set by default in CleanStart manifests  containers:    - name: redis      image: gcr.io/clean-image-build/redis:8.2.2-r0

Available Debug Tools

Debug variant images include these diagnostic tools:

Network

System

Shells

Application-Specific

Some debug images include application-specific tools:

Common Debug Scenarios

Application Won't Start

kubectl debug -it my-pod \  --image=gcr.io/clean-image-build/myapp:v1.0-debug # Check if the binary exists and is executablels -la /app/serverfile /app/server # Try running it manually to see error output/app/server --config /etc/app.conf

Network Connectivity Issues

# DNS resolutiondig my-database.default.svc.cluster.local # TCP connectivitync -zv my-database 5432 # HTTP endpoint checkcurl -v http://my-service:8080/health

Process Memory Dump (.NET)

# Capture a full memory dump of the production processprocdump -ma 1 /tmp/app.dmp # Copy the dump out of the podkubectl cp my-pod:/tmp/app.dmp ./app.dmp

File System Inspection

# See what the production container's filesystem looks likels -la /proc/1/root/ # Check mounted volumesdf -h # Find large filesfind /proc/1/root/data -size +10M -ls

FIPS Debug Variant

For FIPS-compliant environments, use the -fips-debug variant:

kubectl debug -it my-pod \  --image=gcr.io/clean-image-build/redis:8.2.2-fips-debug

This includes the same debug tools but compiled against FIPS 140-3 validated cryptographic modules.

Troubleshooting

"unable to create ephemeral container" -- Your cluster's Kubernetes version may be too old. Ephemeral containers require Kubernetes 1.23+.

"cannot share process namespace" -- The pod spec needs shareProcessNamespace: true. CleanStart generated manifests include this by default. If you wrote the manifest manually, add it to spec.

Can't see production processes -- Ensure --target=<container-name> is specified to join the correct container's namespace. Without it, the debug container may not share the PID namespace correctly on older Kubernetes versions.

Debug image not found -- Not all images have debug variants. The image template must declare debug_packages to generate a debug build. Check your registry for available tags with the -debug suffix.