Building Rust Applications with CleanStart

CleanStart provides production-ready Rust base images (cleanstart/rust) with Rust 1.75 and 1.76 toolchains, Cargo for dependency management, multi-stage static compilation, Linux musl targets for maximum portability, and minimal distroless production images.

Quick Start: A Simple Web Service

Step 1: Create Your Project

cargo new my-rust-appcd my-rust-app

Step 2: Add Dependencies

Edit Cargo.toml:

[package]name = "my-rust-app"version = "1.0.0"edition = "2021" [dependencies]axum = "0.7"tokio = { version = "1", features = ["full"] }serde_json = "1"tracing = "0.1"tracing-subscriber = { version = "0.3", features = ["json"] } [profile.release]opt-level = 3lto = truecodegen-units = 1strip = true

Step 3: Write Your Application

Edit src/main.rs:

use axum::{    extract::Json,    http::StatusCode,    routing::{get, post},    Router,};use serde_json::{json, Value};use std::net::SocketAddr;use tracing::info; #[tokio::main]async fn main() {    // Initialize tracing    tracing_subscriber::fmt()        .with_writer(std::io::stdout)        .json()        .init();     // Build router    let app = Router::new()        .route("/health", get(health_handler))        .route("/api/data", get(data_handler))        .route("/api/echo", post(echo_handler));     let addr = SocketAddr::from(([0, 0, 0, 0], 8080));    info!("Server starting on {}", addr);     let listener = tokio::net::TcpListener::bind(addr)        .await        .expect("Failed to bind");     axum::serve(listener, app)        .await        .expect("Server error");} async fn health_handler() -> Json<Value> {    Json(json!({"status": "healthy", "version": "1.0"}))} async fn data_handler() -> Json<Value> {    info!("Data endpoint called");    Json(json!({        "message": "Hello from CleanStart",        "version": "1.0"    }))} async fn echo_handler(    Json(payload): Json<Value>,) -> (StatusCode, Json<Value>) {    info!("Echo endpoint called");    (StatusCode::OK, Json(payload))}

Step 4: Create Production Dockerfile

# Build stageFROM cleanstart/rust:1.76 AS builder WORKDIR /app # Copy manifest filesCOPY Cargo.toml Cargo.lock* ./ # Create dummy source to cache dependenciesRUN mkdir -p src && \    echo "fn main() {}" > src/main.rs && \    cargo build --release && \    rm -rf src # Copy real source codeCOPY src ./src # Build applicationRUN touch src/main.rs && \    cargo build --release # Production stageFROM gcr.io/distroless/base-debian12 WORKDIR /app # Copy binary from builderCOPY --from=builder /app/target/release/my-rust-app /app/app EXPOSE 8080 ENTRYPOINT ["/app/app"]

Step 5: Build and Run

# Build Docker imagedocker build -t my-rust-app:latest . # Run the containerdocker run --rm -p 8080:8080 my-rust-app:latest # Test the applicationcurl http://localhost:8080/api/datacurl http://localhost:8080/health

Optimized Multi-Stage Build

Reduce build time with dependency layer separation:

# Dependency cache stageFROM cleanstart/rust:1.76 AS planner WORKDIR /appRUN cargo install cargo-chef COPY . .RUN cargo chef prepare --recipe-path recipe.json # Build dependencies stageFROM cleanstart/rust:1.76 AS cacher WORKDIR /appRUN cargo install cargo-chef COPY --from=planner /app/recipe.json recipe.jsonRUN cargo chef cook --release --recipe-path recipe.json # Build application stageFROM cleanstart/rust:1.76 AS builder WORKDIR /app COPY . .COPY --from=cacher /app/target target RUN cargo build --release # Production stageFROM gcr.io/distroless/base-debian12 COPY --from=builder /app/target/release/my-rust-app /app/app EXPOSE 8080ENTRYPOINT ["/app/app"]

This pattern significantly speeds up rebuilds when dependencies don't change.

Testing

Create Tests

Create src/lib.rs for library code:

pub fn add(a: i32, b: i32) -> i32 {    a + b} #[cfg(test)]mod tests {    use super::*;     #[test]    fn test_add() {        assert_eq!(add(2, 2), 4);    }}

Integration Tests

Create tests/integration_test.rs:

#[test]fn test_addition() {    // Integration test example    assert_eq!(2 + 2, 4);}

Test in Docker

FROM cleanstart/rust:1.76 AS test WORKDIR /appCOPY . . RUN cargo test --verbose --release

Run tests: docker build --target test -t my-rust-app:test .

Production Best Practices

Environment Configuration

use std::env; pub struct Config {    pub port: String,    pub log_level: String,    pub database_url: Option<String>,} impl Config {    pub fn from_env() -> Self {        Config {            port: env::var("PORT").unwrap_or_else(|_| "8080".to_string()),            log_level: env::var("LOG_LEVEL").unwrap_or_else(|_| "info".to_string()),            database_url: env::var("DATABASE_URL").ok(),        }    }}

Run with environment variables:

docker run \    -e PORT=8080 \    -e LOG_LEVEL=debug \    -e DATABASE_URL=postgresql://user:pass@db:5432/app \    my-rust-app:latest

Structured Logging

use tracing::{info, error, warn}; fn process_request(req_id: String) {    info!(        request_id = %req_id,        "Processing request"    );     if let Err(e) = perform_operation() {        error!(            request_id = %req_id,            error = %e,            "Operation failed"        );    }}

Graceful Shutdown

use tokio::signal; #[tokio::main]async fn main() {    let app = build_app();     let listener = tokio::net::TcpListener::bind("0.0.0.0:8080")        .await        .unwrap();     let shutdown = async {        signal::ctrl_c()            .await            .expect("Failed to install CTRL+C signal handler");        info!("Shutdown signal received");    };     axum::serve(listener, app)        .with_graceful_shutdown(shutdown)        .await        .unwrap();}

Health Checks

async fn health_handler() -> impl IntoResponse {    let health = serde_json::json!({        "status": "healthy",        "uptime": get_uptime()    });    (StatusCode::OK, Json(health))} async fn ready_handler() -> impl IntoResponse {    match check_dependencies().await {        Ok(_) => (StatusCode::OK, Json(json!({"ready": true}))),        Err(e) => (            StatusCode::SERVICE_UNAVAILABLE,            Json(json!({"ready": false, "error": e.to_string()}))        ),    }}

Performance Optimization

Release Profile

[profile.release]opt-level = 3          # Maximum optimizationlto = true             # Link-time optimizationcodegen-units = 1      # Better optimization (slower compile)strip = true           # Strip debug symbolspanic = "abort"        # Smaller binary

Zero-Cost Abstractions

Rust enables high-level code without runtime overhead:

// This compiles to the same machine code as manual loopslet sum: i32 = (1..=100).sum(); // Iterators are optimized awaylet doubled: Vec<i32> = vec![1, 2, 3]    .iter()    .map(|x| x * 2)    .collect();

Async/Await for Concurrency

Handle thousands of connections efficiently:

async fn handle_connection(socket: TcpStream) {    // Non-blocking I/O    let (reader, writer) = socket.into_split();    // Process...}

Security Best Practices

Use Distroless Images

# Bad: Contains shell and utilitiesFROM cleanstart/rust:1.76 # Good: Minimal distroless imageFROM gcr.io/distroless/base-debian12COPY --from=builder /app/target/release/my-rust-app /ENTRYPOINT ["/my-rust-app"]

Memory Safety

Rust prevents memory leaks and buffer overflows at compile time:

// This won't compile: use after free preventionlet s = String::from("hello");println!("{}", s);  // OKprintln!("{}", s);  // OK - moved value still accessible // Rust prevents this at compile time:let x = vec![1, 2, 3];let y = x;  // x moved to y// println!("{}", x);  // Compile error!

Input Validation

fn validate_email(email: &str) -> Result<(), String> {    if email.len() < 5 || !email.contains('@') {        return Err("Invalid email".to_string());    }    Ok(())} fn process_input(input: String) -> Result<Output, Error> {    validate_email(&input)?;    // Process...}

Dependency Scanning

cargo audit                    # Check for known vulnerabilitiescargo update                   # Update dependenciescargo outdated                 # Show outdated versions

Platform-Specific Builds

Linux musl (Alpine-compatible)

rustup target add x86_64-unknown-linux-muslcargo build --release --target x86_64-unknown-linux-musl

Cross-compilation

FROM cleanstart/rust:1.76 RUN rustup target add x86_64-unknown-linux-gnuRUN rustup target add aarch64-unknown-linux-gnu COPY . .RUN cargo build --release --target x86_64-unknown-linux-gnuRUN cargo build --release --target aarch64-unknown-linux-gnu

Deploying to Cloud Run

# Build and pushgcloud builds submit --tag gcr.io/my-project/my-rust-app # Deploygcloud run deploy my-rust-app \    --image gcr.io/my-project/my-rust-app:latest \    --platform managed \    --region us-central1 \    --allow-unauthenticated \    --memory 256Mi \    --cpu 1 \    --set-env-vars PORT=8080,LOG_LEVEL=info

Image Options

Troubleshooting

Build Issues

Problem: Cargo build very slow; every source change triggers full recompilation Cause: Cargo.toml and source copied together; Docker layer caching defeated; dependencies recompiled on code change Fix: Structure Dockerfile to separate dependency layer from source: COPY Cargo.toml Cargo.lock ./ then RUN cargo build --release (creates artifact cache), then COPY src ./src and final build. For even better caching, use cargo-chef (see optimized multi-stage example in guide). This way, dependency layer only rebuilds if Cargo.toml changes.

Problem: cargo build fails with "package not found" or "failed to fetch" Cause: Dependency not in crates.io or git; internet connectivity issue in Docker; crate version mismatch; git dependency unreachable Fix: Verify dependency exists: cargo search package_name. Check Cargo.toml syntax. In Docker, ensure internet connectivity. For git dependencies, use HTTPS: git = "https://...". Run locally first: cargo build to test. In CI, consider vendoring dependencies: cargo vendor and copy vendor/ directory.

Problem: cargo build --release runs out of memory or takes extreme time Cause: Docker memory limit too low; link-time optimization (LTO) enabled; parallel compilation spawning too many processes; codegen-units=1 with large dependency tree Fix: Increase Docker memory: docker build --memory=4g. For LTO, it's slow but necessary for optimization. Reduce CODEGEN_UNITS to balance: codegen-units = 16 uses less memory but slower (default 16, slower). Or disable LTO for development: lto = false in Cargo.toml for debug builds. Use cargo build --release -j2 to limit parallel jobs. Enable BuildKit inline cache for layer caching: docker buildx build --cache-from=type=inline.

Problem: Binary size very large (50MB+) when should be small (5-10MB) Cause: Debug symbols included; strip = true not set; unnecessary dependencies; not using release profile Fix: Add to Cargo.toml: [profile.release] strip = true. Build with: cargo build --release. Verify: ls -lh target/release/my-app and file target/release/my-app should show "stripped". Check dependencies: cargo tree to identify large dependencies, consider alternatives. Use cargo bloat --release to analyze what's consuming space.

Problem: Incremental compilation cache not working across Docker builds Cause: /target directory not mounted as cache layer; cargo incremental compilation default in debug only; separate build layers recreate state Fix: For Docker BuildKit (docker buildx), use cache mount: RUN --mount=type=cache,target=/app/target cargo build --release. For older Docker, accept longer builds or use cache layer mount. Incremental compilation is disabled in release mode by default (intended), so full rebuild is expected for release builds.

Problem: Cross-compilation fails - "unknown target x86_64-unknown-linux-musl" Cause: Target architecture not installed; cross-compile tool missing; wrong RUSTFLAGS for target Fix: Install target: rustup target add x86_64-unknown-linux-musl. In Dockerfile: RUN rustup target add x86_64-unknown-linux-musl && cargo build --release --target x86_64-unknown-linux-musl. For ARM64: rustup target add aarch64-unknown-linux-gnu. Use cross tool for easy setup: cargo install cross then cross build --target aarch64-unknown-linux-gnu --release.

Problem: Native library linking fails during build; undefined reference errors Cause: C library not installed in builder; linker misconfigured; musl libc incompatibility with C library Fix: For musl target (x86_64-unknown-linux-musl), most C libraries need special compilation. Use rustls for TLS (pure Rust): features = ["rustls-tls"] instead of openssl. For system libraries, use glibc target instead: x86_64-unknown-linux-gnu. Install build tools in builder: RUN apt-get install -y build-essential pkg-config. Check library docs for Rust-specific build instructions.

Problem: Cargo.lock not committed; build results vary or dependency hell in CI Cause: Cargo.lock ignored in .gitignore; library crate without lockfile (correct); binary crate needs lockfile for reproducibility Fix: For binary crates (applications), commit Cargo.lock: git add Cargo.lock. For libraries, don't commit Cargo.lock (libraries depend on host app's lockfile). Ensure Cargo.lock in Docker build context. In production CI, use cargo build --locked to ensure exact versions from lockfile.

Runtime Issues

Problem: Binary won't execute in distroless image - "file not found" or "cannot execute binary" Cause: Dynamic linking to glibc (musl target not built); binary format incompatibility; stripped too aggressively; missing dynamic loader Fix: Build with musl target for static linking: cargo build --release --target x86_64-unknown-linux-musl. Verify static: ldd target/.../app should show "not a dynamic executable". If using glibc target, use CleanStart/Go image or full Ubuntu base, not distroless. For musl, file output should show "statically linked".

Problem: Signal handling doesn't work; application doesn't gracefully shutdown on SIGTERM Cause: Signal handler not registered; tokio runtime issue; cleanimg-init not forwarding signals properly Fix: Register signal handler (see guide example with tokio::signal). Test: docker kill -s TERM container-id should trigger shutdown in logs. Ensure log output shows "Shutdown signal received". Verify graceful_shutdown is awaited in axum/actix-web setup. With cleanimg-init as PID 1, signals forwarded to direct child (Rust binary).

Problem: DNS resolution fails - "no such host" or "Name or service not known" Cause: /etc/resolv.conf missing in distroless; pure Rust DNS resolver having issues; network namespace isolation Fix: Copy resolv.conf from builder: COPY --from=cleanstart/rust /etc/resolv.conf /etc/. Rust's trust-dns crate should work without system resolver. If issues, ensure tokio runtime has network access: test with tokio::net::lookup_host(). In Kubernetes, DNS should be automatic (pod inherits /etc/resolv.conf from cluster DNS). For local Docker, use --dns 8.8.8.8 if needed.

Problem: TLS/HTTPS connections fail - "certificate signed by unknown authority" Cause: CA certificates missing in distroless; rustls not finding system certificates; custom CA not mounted Fix: Copy CA bundle from builder: COPY --from=cleanstart/rust /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/. Use rustls (pure Rust TLS) to avoid C library dependency: features = ["rustls-tls"] in Cargo.toml. For custom CAs, mount as ConfigMap in Kubernetes. Verify: test TLS connection in container with custom curl (if available) or via HTTP endpoint checking.

Problem: Panic in production - application crashes with backtrace, no debugging possible Cause: No shell to debug in distroless; panic output not captured; stack trace unhelpful Fix: Ensure panic output logged to stdout: RUST_BACKTRACE=1 env var. In code, use unwrap_or_else() instead of unwrap() to provide fallback. Log full context before risky operations. For production, set panic = "abort" in release profile for predictable failure. Enable core dumps if needed: ulimit -c unlimited. Monitor logs: panic output goes to stdout (captured by Docker logs).

Problem: Memory grows unbounded over time; suspected memory leak Cause: Event listeners or futures not cleaned up; unbounded channels; caches not evicted; Arc cycles; string/byte allocations not freed Fix: Use memory profiler: valgrind or custom instrumentation. In code, verify drops: #[derive(Debug)] to ensure cleanup logic visible. Use bounded channels: tokio::sync::mpsc::channel(1000) not unbounded channel(). Implement TTL caches: evict old entries periodically. Use std::rc::Weak for potential cycles. Profile with heaptrack if available in builder.

Problem: Application crashes with "stack overflow" or "thread stack overflowed" Cause: Infinite recursion; unbounded async recursion; stack not sized for workload; deep call chains in dependencies Fix: Identify recursive function: use clippy lint disallowed_names to catch common recursion issues. For explicit OS threads, increase stack size if needed: thread::Builder::new().stack_size(8 * 1024 * 1024).spawn(...). For Tokio async tasks, stack size is configured via the runtime: #[tokio::main(worker_threads = 4)] or tokio::runtime::Builder::new_multi_thread().worker_threads(4).build(). Note: stack size tuning applies to explicit threads only; async runtime handles its own stack management. In containers, stack size from kernel defaults (usually sufficient). Check dependencies for deep call chains in hot paths.

Problem: Application can't write to /tmp or application directories; permission denied Cause: /tmp read-only in read-only filesystem; application running as wrong UID (65532); directory ownership issue Fix: For read-only root, mount /tmp as tmpfs in Kubernetes: volumeMounts: [{name: tmp, mountPath: /tmp}]. Create writable app directory in Dockerfile: RUN mkdir -p /app/data && chmod 777 /app/data. For UID 65532 (non-root in CleanStart), ensure directories are world-writable or owned by that UID. Test: docker run my-app touch /tmp/test should succeed.

Problem: Container startup very slow; binary initialization taking 10+ seconds Cause: Async initialization blocking startup; heavy computation on startup path; I/O operations during init; TLS handshakes Fix: Defer non-critical initialization to lazy loading. Profile startup: add timing logs at key points. For database connections, use connection pooling initialized on first request, not startup. Move TLS cert loading to lazy: load on first use. For true startup optimization, consider GraalVM native-image (Rust already near-instant startup, so likely code issue).

Performance & Optimization Issues

Problem: Release binary slower than expected despite optimizations; CPU usage high Cause: Optimizations not applied correctly; release profile wrong; dependencies causing slowness; suboptimal algorithms Fix: Verify release build used: cargo build --release. Check profile settings: opt-level = 3, lto = true. Profile with flamegraph: cargo install flamegraph then cargo flamegraph --release. Identify hot functions and optimize. Consider disabling LTO temporarily to verify it's not culprit. Profile allocation: use jemalloc allocator: [profile.release] default-allocator = true or use jemalloc_sys crate.

Problem: Binary crashes under high concurrency; "too many open files" errors Cause: File descriptor limit; unbounded connection pools; not respecting system ulimit Fix: Increase ulimit in container: docker run --ulimit nofile=65535:65535 my-app. In code, limit concurrent connections: Arc<Semaphore>::new(100) for max concurrent tasks. Use connection pooling with bounded size. In Kubernetes, set SysCtl: sysctls: [{name: fs.file-max, value: "65535"}]. Monitor open fds: lsof -p <pid> | wc -l.

Problem: Jemalloc allocator not improving performance; memory fragmentation suspected Cause: Jemalloc default settings not tuned; workload doesn't benefit from jemalloc; configuration error Fix: Use jemalloc: add jemallocator crate and set #[global_allocator] static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;. Tune with env vars: MALLOC_CONF=background_thread:true,metadata_thp:auto. Profile with malloc stats: MALLOC_CONF=stats_print:true env var. For specific workloads, system allocator may be sufficient (prefer simpler default unless proven issue).

Problem: Container doesn't respect CPU limits; uses more CPU than allocated Cause: Rust doesn't natively respect cgroup CPU limits (unlike JVM); thread count not limited; compute-bound operations unbounded Fix: Limit thread pool explicitly: num_cpus::get() respects cgroups in newer Rust. Or hardcode: RAYON_NUM_THREADS=2 for rayon. For tokio, runtime threads auto-scale but can be bounded: tokio::runtime::Builder::new_multi_thread().worker_threads(2).build(). Monitor CPU: docker stats. If still excessive, profile to find compute-bound hot spots.

CleanStart Production Hardening

Read-Only Root Filesystem

Enforce immutable infrastructure with read-only root for Rust applications:

apiVersion: v1kind: Podmetadata:  name: rust-appspec:  containers:  - name: app    image: cleanstart/rust:1.76    securityContext:      readOnlyRootFilesystem: true    volumeMounts:    - name: tmp      mountPath: /tmp    - name: app-logs      mountPath: /app/logs  volumes:  - name: tmp    emptyDir: {}  - name: app-logs    emptyDir: {}

Shell-Less ENTRYPOINT

Remove shell for reduced attack surface. Update your Dockerfile:

FROM cleanstart/rust:1.76 AS builder WORKDIR /appCOPY Cargo.* .RUN cargo fetchCOPY . .RUN cargo build --release --target x86_64-unknown-linux-musl FROM cleanstart/rust:1.76 WORKDIR /appCOPY --from=builder /app/target/x86_64-unknown-linux-musl/release/app . # Declarative Image Builder: # Use cleanimg-init as PID 1ENTRYPOINT ["/cleanimg-init", "--"]CMD ["./app"]

cleanimg-customize: Rust Optimization

Inject production optimization flags and environment variables:

FROM cleanstart/rust:1.76 AS builder WORKDIR /appCOPY Cargo.* .RUN cargo fetchCOPY . .RUN cargo build --release --target x86_64-unknown-linux-musl FROM cleanstart/rust:1.76 WORKDIR /appCOPY --from=builder /app/target/x86_64-unknown-linux-musl/release/app . # Set Rust runtime for productionENV RUST_LOG=info RUST_BACKTRACE=1 ENTRYPOINT ["/cleanimg-init", "--"]CMD ["./app"]

Security Context

Complete Kubernetes securityContext for hardened Rust containers:

apiVersion: apps/v1kind: Deploymentmetadata:  name: rust-appspec:  template:    spec:      securityContext:        fsGroup: 65532        seccompProfile:          type: RuntimeDefault      containers:      - name: app        image: cleanstart/rust:1.76        securityContext:          allowPrivilegeEscalation: false          readOnlyRootFilesystem: true          runAsNonRoot: true          runAsUser: 65532          runAsGroup: 65532          capabilities:            drop:            - ALL        resources:          requests:            cpu: 100m            memory: 32Mi          limits:            cpu: 500m            memory: 256Mi

Next Steps

Advanced Rust Deployment, Rust Performance Tuning, and Kubernetes Deployment.