What Is a Hermetic Build?

A hermetic build is a build process that operates with complete transparency and isolation. The term "hermetic" refers to a sealed container—in this context, a build environment that is completely sealed from external influences and whose inputs and outputs are completely documented.

A hermetic build process declares all inputs upfront, establishing a complete inventory of what will go into the final artifact before any compilation or assembly begins. This includes source code, every package dependency, and all configuration parameters. Rather than fetching dependencies during the build—when attackers could potentially intercept requests or serve malicious packages—the process fetches and verifies all dependencies before the build environment is sealed. Once sealed, the build environment runs in complete network isolation: no external downloads occur during the build, no package manager can pull in surprise updates, and no external service can influence what gets compiled. The result is a reproducible output: given the same inputs, the same build process always produces an identical artifact. Finally, the entire process generates cryptographic proof—a build provenance document—that cryptographically proves exactly what went into the image and exactly how it was built.

In essence: A hermetic build proves that the image you have today is byte-for-byte identical to what would be produced by running the same build again tomorrow, and it provides cryptographic evidence to support that claim.

Why Hermetic Builds Matter

Traditional (Non-Hermetic) Build

# DockerfileFROM ubuntu:22.04RUN apt-get updateRUN apt-get install -y python3 python3-pipRUN pip install fastapi uvicorn

This approach appears clean and maintainable, but it creates multiple critical problems. When apt-get update runs, it downloads the latest available versions of packages from the Ubuntu repositories at the moment the build executes. If you rebuild this same Dockerfile in six months, the packages that get installed will be different. Similarly, when pip install fastapi executes, it resolves to whatever version of FastAPI is currently available in PyPI at build time. The exact version is unpredictable and depends on the timing of the build.

The fundamental problems are clear. apt-get update downloads latest packages with no version control, so rebuilds are unpredictable. pip install fastapi resolves to the latest version available at build time, not a specific declared version. If you rebuild this image in six months, you will get completely different packages. There is no record of what was actually installed, making the build non-reproducible. And most importantly, build output is not reproducible—the same Dockerfile run at different times produces different images.

Hermetic Build (CleanStart)

A hermetic build approach using CleanStart's configuration language is explicitly declarative about every input:

name: python-fastapipython_version: 3.12.1packages:  fastapi: 0.104.1  uvicorn: 0.24.0build_type: hermetic

This configuration declares exactly which versions of Python, FastAPI, and Uvicorn should be included in the image. The benefits are transformative. All versions are pinned upfront before the build begins, eliminating any unpredictability. No network calls occur during the build itself—all dependencies are pre-fetched in the dependency resolution phase and cached locally. If you rebuild this exact same configuration six months later, the result is a byte-for-bit identical image, which you can verify by comparing SHA-256 hashes. The complete build provenance is recorded in a cryptographic format that documents every step. Finally, the image is completely verifiable and auditable—anyone can cryptographically verify the build process and detect any tampering.

How Hermetic Builds Work in CleanStart

The following diagram illustrates the hermetic build process from configuration through reproducible output:

graph TD    A["Build<br/>Configuration<br/>YAML"] -->|All Inputs<br/>Declared| B["Dependency<br/>Resolution"]     B -->|Fetch| C["Python 3.12.1<br/>Source"]    B -->|Fetch| D["FastAPI 0.104.1<br/>PyPI"]    B -->|Fetch| E["Uvicorn 0.24.0<br/>PyPI"]     C -->|Verify<br/>Signature| C1["GPG Check<br/>Valid"]    D -->|Verify<br/>Hash| D1["SHA-256<br/>Match"]    E -->|Verify<br/>Hash| E1["SHA-256<br/>Match"]     C1 -->|Store| F["Lockfile<br/>All Dependencies<br/>Pinned with Hashes"]    D1 -->|Store| F    E1 -->|Store| F     F -->|Input| G["Hermetic Build<br/>Environment<br/>Network: OFF"]     G -->|Compile| H["Python<br/>CPython"]    G -->|Install| I["FastAPI<br/>Uvicorn<br/>Pydantic"]     H -->|Generate| J["SBOM"]    I -->|Generate| J     J -->|Sign| K["Cosign<br/>Signature"]     G -->|Generate| L["Build<br/>Provenance<br/>SLSA Level 4"]     K -->|Output| M["Signed<br/>Image"]    L -->|Output| M     M -->|Archive| N["Image<br/>Registry"]     O["Same Config<br/>6 Months Later"] -->|Build| P["Hermetic Build<br/>Network: OFF"]     P -->|Same Inputs| Q["Identical<br/>Output<br/>Bit-for-Bit"]     Q -->|Verify| R["Image Hash<br/>Matches<br/>Reproducible"]     style A fill:#ccffcc    style F fill:#99ccff    style G fill:#99ccff    style M fill:#99ff99    style R fill:#99ff99

Stage 1: Dependency Resolution

The process begins by reading your configuration, which specifies Python 3.12.1, FastAPI 0.104.1, and Uvicorn 0.24.0.

The dependency resolution engine then executes a sequence of actions. It fetches Python 3.12.1 from the official CPython repository and verifies the GPG signature. It fetches FastAPI 0.104.1 from PyPI and verifies its SHA-256 hash. It resolves transitive dependencies automatically, pinning versions for Pydantic, Starlette, and Typing-extensions. It generates a lockfile that documents all 50+ direct and transitive dependencies with their exact versions and hashes.

The key principle is this: nothing is downloaded during the actual build. All dependencies are fetched and verified in the dependency resolution phase. The lockfile ensures reproducibility and allows auditing of every component before compilation begins.

Stage 2: Hermetic Compilation

The build environment runs completely isolated from the network. Inputs include the CPython source code, all pre-fetched dependencies cached locally, and build configuration specifying compilation flags.

Build steps execute sequentially in the isolated environment. The process runs ./configure with flags like --prefix=/usr and --enable-fips to configure FIPS compliance. It executes make -j8 to compile using 8 parallel CPU cores. It runs make install to install the compiled binaries. It executes strip to remove debug symbols and reduce binary size.

Outputs from this stage include Python 3.12.1 compiled binary with FIPS enabled, complete build log documenting every compilation step, and metadata about which compilation flags were applied.

The network isolation guarantee is absolute: no curl, wget, or package manager is available during the build. Every input was pre-fetched and verified, preventing last-minute downloads or supply chain tampering.

Stage 3: Dependency Installation

Still in the network-isolated environment, the build system installs all pre-fetched Python packages. It installs FastAPI 0.104.1 from cached wheels (not source, avoiding recompilation). It installs all 50+ transitive dependencies in dependency order. It verifies the SHA-256 hash of every wheel against the lockfile.

All packages are installed to /usr/local/lib/python3.12/site-packages/.

The hash verification guarantee is ironclad: every wheel must match the hash recorded in the lockfile. If a single byte differs, the build fails immediately. This prevents package tampering between the dependency resolution phase and installation.

Stage 4: Build Artifacts

Upon successful completion, the build system produces multiple artifacts. These include the OCI Image (tar.gz format) containing the compiled runtime, build log (complete transcript of every compilation step), SBOM (software bill of materials) documenting all 50+ dependencies, compilation flags (metadata recording exactly how the code was compiled), source manifest (documenting which source repositories and versions were used), and checksums (SHA-256 hashes of all output files).

The reproducibility guarantee means these artifacts enable perfect image reproduction. Given the same configuration, sources, and dependencies, you can rebuild this exact image bit-for-bit 10 years from now. The complete build log provides the evidence needed to do so.

SLSA Level 4 Provenance

SLSA (Supply-chain Levels for Software Artifacts) is a framework for supply chain security. SLSA Level 4 is the highest level and requires hermetic builds (network isolated), source provenance (link to source code repo and commit), build environment control (isolated, auditable build runner), and artifact verification (outputs signed and timestamped).

CleanStart achieves all four requirements and generates a SLSA v1.0 provenance document for every image.

SLSA Provenance Document

{  "version": "1.0",  "subject": [    {      "name": "python-fastapi:3.12.1",      "digest": {        "sha256": "abcd1234efgh5678ijkl9012mnop3456..."      }    }  ],  "predicate": {    "builder": {      "id": "https://cleanstart.io/builders/hermetic-v1"    },    "sourceUri": "https://github.com/python/cpython/releases/tag/v3.12.1",    "buildType": "https://cleanstart.io/buildType/hermetic-container-build-v1",    "buildConfig": {      "hermetic": true,      "networkIsolated": true,      "reproducible": true,      "fipsEnabled": true,      "architectures": ["amd64", "arm64"],      "compilationFlags": "-O2 -fstack-protector-all -D_FORTIFY_SOURCE=2"    },    "buildInvocation": {      "configSource": {        "uri": "https://github.com/myorg/myrepo/blob/main/images/python-fastapi.yaml",        "digest": {          "sha256": "xyz789..."        }      },      "parameters": {        "python_version": "3.12.1",        "fastapi_version": "0.104.1",        "uvicorn_version": "0.24.0"      },      "environment": {        "os": "linux",        "arch": "amd64",        "buildId": "build-20240115-abcd1234",        "timestamp": "2024-01-15T10:30:00Z"      }    },    "materials": [      {        "uri": "https://github.com/python/cpython.git",        "digest": {          "sha256": "abc123..."        }      },      {        "uri": "pkg:npm/fastapi@0.104.1",        "digest": {          "sha256": "def456..."        }      }    ],    "byproducts": {      "buildLog": "[complete build transcript]",      "buildDuration": "482 seconds",      "testResults": "78 tests passed"    },    "completeness": {      "parameters": true,      "environment": true,      "materials": true    },    "reproducible": true  }}

What does this prove? The build was hermetic (network isolated). Source came from GitHub (specific commit/tag). Build environment was controlled (CleanStart builder). Exact versions of dependencies were used. Compilation flags were applied. Build log is complete and auditable. All inputs and outputs are documented.

Reproducible Builds

Bit-for-Bit Reproducibility

Consider this scenario: you build a Python image today. Six months from now, you rebuild it with the same config.

With the traditional approach, the result is different. pip install fastapi resolves to a different version. Ubuntu packages are updated. Compiler version changes. The result: non-reproducible.

With the CleanStart hermetic approach, the result is identical. Python 3.12.1 source is always the same. FastAPI 0.104.1 wheel is always the same. All dependencies are pinned in lockfile. Compiler is pinned. The result: bit-for-bit identical.

Verification

To verify reproducibility:

# Build 1 (today)clnstrt-cli build --config python-fastapi.yaml --output image1.tar.gzSHA1=$(sha256sum image1.tar.gz) # ... 6 months pass ... # Build 2 (6 months later)clnstrt-cli build --config python-fastapi.yaml --output image2.tar.gzSHA2=$(sha256sum image2.tar.gz) # Compareif [ "$SHA1" == "$SHA2" ]; then  echo "Builds are identical (reproducible)"else  echo "Builds differ (not reproducible)"fi

With CleanStart: builds are always identical (assuming same config and source availability).

Network Isolation Benefits

Security Advantage

A network-isolated build cannot download malware during build, connect to a compromised mirror, leak secrets to external systems, or download unexpected versions of dependencies. All dependencies come from a pre-verified cache; nothing surprises you.

Example: Supply Chain Attack Prevention

Consider this scenario: an attacker compromises the npm registry and injects malicious JavaScript into the fastapi package.

With the traditional build: downloads from npm during build, which includes malicious code.

With the hermetic build: pre-fetches from npm, verifies hash, then builds offline. If attacker modifies package, hash check fails and build aborts. No connection to npm during build means no way for attacker to intercept.

Compilation Flags and Hardening

CleanStart applies security-hardening compilation flags by default. The flag -fstack-protector-all protects against stack buffer overflows with canaries before function returns. -D_FORTIFY_SOURCE=2 adds bounds checking to string operations (strcpy becomes bounds-checked version). -fPIE enables position-independent executable for ASLR support (address randomization). -fno-semantic-interposition optimizes based on security assumptions (faster and secure). -Wl,-z,relro makes relocation sections read-only to prevent GOT overwrite attacks. -Wl,-z,now resolves symbols at load time (not lazy) to prevent symbol hijacking.

The result: binaries are hardened against common attack vectors before any vulnerability scanning.

Multi-Architecture Hermetic Builds

CleanStart supports native compilation for multiple architectures, ensuring optimal performance and eliminating cross-compilation issues.

AMD64 Hermetic Build

Execution occurs on dedicated AMD64 builder nodes. The process compiles Python 3.12.1 using the native AMD64 GCC toolchain. It installs FastAPI and all dependencies using AMD64-specific wheels. Output: python-fastapi:3.12.1-amd64.

ARM64 Hermetic Build

Execution occurs on dedicated ARM64 builder nodes. The process compiles Python 3.12.1 using the native ARM64 GCC toolchain. It installs FastAPI and all dependencies using ARM64-specific wheels. Output: python-fastapi:3.12.1-arm64.

The key principle is this: no cross-compilation is used. Native compilation on actual hardware eliminates architecture-specific toolchain bugs and ensures binaries are optimized for their target architecture.

Multi-Architecture Manifest

{  "schemaVersion": 2,  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",  "manifests": [    {      "mediaType": "application/vnd.docker.container.image.v1+json",      "size": 107925051,      "digest": "sha256:amd64-digest-here",      "platform": {        "architecture": "amd64",        "os": "linux"      }    },    {      "mediaType": "application/vnd.docker.container.image.v1+json",      "size": 112841920,      "digest": "sha256:arm64-digest-here",      "platform": {        "architecture": "arm64",        "os": "linux"      }    }  ]}

The benefit: a single tag (python-fastapi:3.12.1) automatically pulls the correct architecture image.

Build Performance

Typical Build Times

The overhead is 15-25% additional time for hermetic isolation, dependency verification, and provenance generation. The justification is clear: the security and auditability benefits far outweigh the small time overhead.

Parallel Multi-Architecture Builds

CleanStart builds AMD64 and ARM64 variants in parallel, optimizing total build time. The build timeline works like this: minutes 0-5 involve dependency resolution (once, shared across both architectures). Minutes 5-15 have AMD64 build running in parallel with ARM64 build on separate hardware nodes. Minutes 15-18 involve testing and verification (78-test suite runs on both variants simultaneously). Minutes 18-20 involve pushing both images to registries and creating multi-architecture manifest.

Total end-to-end time: 20 minutes for dual-architecture build (not 40 minutes as it would be with sequential builds). Parallel native compilation on dedicated hardware significantly reduces total time without sacrificing quality.

Audit Trail and Forensics

Complete Build Audit Trail

Every hermetic build generates multiple artifacts. The build log contains a complete transcript of every command, output, and error. SLSA provenance is cryptographically signed proof of build process. Dependency manifest documents every package and version used. Compilation flags records exactly which flags were used during compilation. Test results show 78-test suite results for verification.

Forensic Example

Question: "How do I know this image has not been tampered with?"

Answer: First, verify image signature with cosign verify image:tag. Second, retrieve SLSA provenance with cosign download attestation image:tag. Third, inspect provenance to see all inputs, build steps, and outputs documented. Fourth, verify reproducibility by re-running build with same config and comparing checksums. Fifth, the result is complete proof of image integrity.

Compliance and Governance

Audit-Ready by Default

Hermetic builds satisfy requirements from NIST (supply chain security, reproducibility), SOC 2 (build integrity, complete audit trail), SLSA (Level 4 certification, highest level), CISA (supply chain security guidelines), and PCI-DSS (secure build processes).

Policy Enforcement

CleanStart enforces policies to ensure quality. No unpinned versions are allowed. All dependencies must be from official sources. Security compilation flags are required. Build provenance is required. Test suite must pass.

Next Steps

Understand the build pipeline at Image Construction Overview. Learn how to configure images at YAML Image Configuration. Explore dependency intelligence at Dependency Intelligence. Start building with the quick start guide.

In Practice

Hermetic builds plus SLSA Level 4 equals provable security, not assumed security.

You can prove your image is what you think it is, built how you intended it to be built, and free from tampering. This is not a feature—it is a requirement for modern supply chain security.