Building Go Applications with CleanStart

CleanStart provides optimized Go base images (cleanstart/go) pre-configured for container development. These images include Go 1.21 and 1.22 toolchains for cutting-edge language features, support fast static binary compilation to create minimal containers, provide minimal distroless production images for ultra-small runtime footprints, include full CGO support for applications that need to call C libraries, and have multi-stage optimization built-in to enable efficient development and production image separation.

The Go application compilation and containerization workflow includes building from source (main.go, go.mod), downloading dependencies, verifying checksums, compiling to static binary, and outputting a minimal 5 MB ELF binary. The production stage uses either distroless/base or cleanstart-minimal, sets up non-root user (UID 65532), enforces no shell and read-only FS, exposes port 8080, and creates a secure 5 MB container image.

Quick Start: A Simple HTTP Server

Step 1: Initialize Your Project

mkdir my-go-appcd my-go-appgo mod init github.com/example/my-go-app

Step 2: Write Your Application

Create main.go with a Response struct, health handler, API handler, and main function. The main function sets up JSON logging, registers handlers at /health and /api/data, creates an HTTP server with timeouts, implements graceful shutdown on SIGTERM/SIGINT, and starts the server.

Step 3: Add Dependencies

go mod tidy

Step 4: Create Production Dockerfile

The build stage uses cleanstart/go:1.22, copies go.mod and go.sum, downloads dependencies, copies source code, and builds static binary with -ldflags="-w -s" to strip debug info. The production stage uses gcr.io/distroless/base-debian12, copies built binary, and runs the application.

Step 5: Build and Run

# Build Docker imagedocker build -t my-go-app:latest . # Run the containerdocker run --rm -p 8080:8080 my-go-app:latest # Test the applicationcurl http://localhost:8080/api/datacurl http://localhost:8080/health

Understanding the Build Strategy

Go compiles to static binaries, making it ideal for lightweight containers. The build stage compiles code in the heavy builder image. The production stage uses tiny distroless image with only the binary. This reduces final image size from 500MB+ to 10-50MB.

Multi-Stage with Debugging

For development and debugging, keep a debug image. The debug stage uses cleanstart/go:1.22, installs delve debugger, and exposes port 2345 for debugging. Build debug image with docker build --target debug.

Testing

Create Tests

Create main_test.go with test functions for HealthHandler and APIHandler using httptest.

Test in Docker

Create a test stage in Dockerfile that runs tests. Build test image with docker build --target test.

Production Best Practices

Environment Configuration

Load configuration from environment variables with fallback defaults for PORT, LOG_LEVEL, and DATABASE_URL.

Run with environment variables: docker run -e PORT=8080 -e LOG_LEVEL=debug -e DATABASE_URL=...

Structured Logging

Use slog package for JSON logging with context, method, path, and status.

Graceful Shutdown

Listen for SIGTERM and SIGINT signals, close the server with timeout, and log the shutdown.

Health Checks

Implement /health endpoint checking dependencies and returning appropriate status.

Performance Optimization

Build Flags

Remove debug info with -ldflags="-w -s", embed version info with -X main.Version=1.0.0, and strip symbols for smaller binaries.

Caching Dependencies

Structure Dockerfile to maximize layer caching by copying go.mod and go.sum before source code.

Async and Concurrency

Go handles thousands of concurrent connections through lightweight goroutines.

Security Best Practices

Use Distroless Images

Never include shell or utilities in production. Use distroless/base-debian12 instead of full images.

Input Validation

Validate input length and format to prevent injection attacks.

Dependency Scanning

Use nancy for vulnerability scanning and keep dependencies updated.

Deploying to Cloud Run

Build and push to GCR, then deploy to Cloud Run with resource limits and environment configuration.

Image Options

Advanced: CGO and Native Libraries

When you need C libraries, build with CGO_ENABLED=1 but use appropriate base image.

Troubleshooting

Build Issues

Problem: go mod download fails - "no matching versions for query"

Cause: Cached go.mod from local doesn't match Docker environment, indirect dependencies missing, or version constraints too strict

Fix: Run go mod tidy locally first, ensure RUN go mod download happens before COPY of source, add -x flag for verbose output, cache the go mod download by separating COPY and RUN commands

Problem: CGO_ENABLED=0 not set; binary won't run in distroless/minimal images

Cause: CGO_ENABLED=1 links C libraries dynamically; distroless lacks libc

Fix: Always use CGO_ENABLED=0 GOOS=linux go build for container builds, verify with ldd app (should show "not a dynamic executable"), use full image if you need C libraries

Problem: Build takes much longer after code changes

Cause: Go modules not cached as separate layer, COPY . . before go mod download, binary rebuild triggers all compilation

Fix: Structure Dockerfile for maximum caching with COPY go.mod/go.sum before COPY source

Problem: Binary larger than expected (50MB+ when should be 5MB)

Cause: Debug symbols included, not using -ldflags="-w -s", binary not stripped, unnecessary dependencies linked

Fix: Use go build -ldflags="-w -s", verify size with ls -lh app, use go tool objdump to analyze

Problem: Build OOM or takes extreme time

Cause: Entire compilation in single layer, no dependency caching, too many build targets, Docker memory limit low

Fix: Use RUN --mount=type=cache,target=/go/pkg/mod to cache, increase Docker memory, build single target, increase GOGC

Problem: Cross-compilation fails

Cause: GOOS/GOARCH set incorrectly, CGO enabled for cross-compile

Fix: For Linux x86_64: GOOS=linux GOARCH=amd64, for ARM64: GOOS=linux GOARCH=arm64, use CGO_ENABLED=0 for cross-compilation, verify with file app

Runtime Issues

Problem: Binary won't start in minimal/distroless image - missing dependencies

Cause: CGO enabled but distroless lacks libc, DNS/TLS files missing, timezone data missing

Fix: Use CGO_ENABLED=0 for distroless, copy resolver config and CA bundle for DNS/TLS, copy timezone data if needed

Problem: x509: certificate signed by unknown authority when making HTTPS requests

Cause: CA certificates missing in distroless, wrong certificate bundle path

Fix: Copy CA certs from builder: COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

Problem: DNS resolution fails ("no such host")

Cause: /etc/resolv.conf missing or empty, libc.musl issues with Go net package, Kubernetes DNS not accessible

Fix: Ensure /etc/resolv.conf exists, copy from builder if needed, verify Go's pure resolver works

Problem: Signal handling doesn't work; SIGTERM not caught

Cause: Signals not forwarded by PID 1, cleanimg-init not properly forwarding, Go runtime issue

Fix: Go handles SIGTERM by default, verify signal registration, test with docker kill -s TERM

Problem: Timezone data missing; time.Now() shows wrong time

Cause: /usr/share/zoneinfo not in distroless, binary linked with CGO but timezone files not available

Fix: Go 1.20+ statically includes tz data, copy zoneinfo if needed, use UTC (recommended for containers)

Problem: Application can't write to /tmp or logs; permission denied

Cause: /tmp mounted as read-only, wrong UID, read-only root filesystem enforcement

Fix: Create writable app directory, mount emptyDir volumes, run as non-root user (65532)

Problem: Custom configuration or environment variables not working

Cause: Env vars not passed to container, code doesn't read env vars, config file path wrong

Fix: Pass at runtime with -e, verify code reads env vars with os.Getenv(), mount ConfigMap for read-only root

Problem: Health check hangs or always fails

Cause: HTTP port not listening, timeout too short, health endpoint not returning 200, nc/curl not available

Fix: Implement lightweight /health endpoint, test from outside container, use orchestrator-level probes

Performance Issues

Problem: Binary performs poorly; startup slow

Cause: Synchronous initialization blocking startup, database connections on boot, file I/O on startup path

Fix: Move non-critical init to lazy initialization, measure startup time, profile with pprof

Problem: Memory usage grows over time; memory leak suspected

Cause: Goroutine leaks, unclosed channels, resource not freed in defer, circular references

Fix: Check goroutine count with pprof, track allocation with runtime.ReadMemStats(), profile heap, use defer cleanup

Problem: Container killed unexpectedly; resource limits not respected

Cause: Go's GOMAXPROCS not respecting cgroup limits (pre-1.19), memory allocator not container-aware, excessive goroutines

Fix: Go 1.19+ respects cgroup limits automatically, set GOMAXPROCS explicitly if needed, limit goroutines, monitor resource usage

Problem: Application unresponsive under load

Cause: Default pool sizes too small, too many goroutines per request, lock contention, DNS lookups blocking

Fix: Profile under load with pprof, check CPU time, use connection pooling, limit concurrency, cache DNS

CleanStart Production Hardening

Read-Only Root Filesystem

Enforce immutable infrastructure by setting readOnlyRootFilesystem: true and mounting writable volumes for /tmp and /app/logs.

Shell-Less ENTRYPOINT

Remove shell for minimal attack surface. Use cleanimg-init as PID 1 with CMD as the application.

cleanimg-customize: Go Runtime Configuration

Inject Go-specific memory and garbage collection settings with GOGC and GOMAXPROCS environment variables.

Security Context

Complete Kubernetes securityContext for hardened Go containers including dropALL capabilities, non-root user (65532), seccomp default profile, and resource limits.

Next Steps

Advanced Go Deployment, Go Performance Tuning, and Kubernetes Deployment.