Generating in-toto Attestations for Build Steps
In-Toto is a framework for securing the integrity of the software supply chain by creating verifiable links between software artifacts and the steps that produced them. Each step in the supply chain (code review, build, test, sign) produces a "link" document that records what was done and what changed.
CleanStart uses In-Toto to create verifiable supply chain links, enabling downstream verification that artifacts went through all required steps and were not tampered with.
In-Toto Concepts
The following diagram illustrates the in-toto supply chain link verification flow:
graph TB A["Source Code<br/>Commit"] -->|Step 1| B["Code Review<br/>Link"] B -->|Signed| C["Review Link<br/>Input: src/*<br/>Output: approved<br/>Reviewer Signature"] C -->|Required| D{All Steps<br/>Complete?} D -->|No| E["Review Link<br/>Locked<br/>Pending"] D -->|Yes| F["Step 2: Build"] F -->|Record| G["Build Link<br/>Input: src/* + Dockerfile<br/>Output: image:1.0.0<br/>Builder Signature"] G -->|Check| H["Verify Build<br/>Happened"] H -->|Yes| I["Step 3: Test"] I -->|Record| J["Test Link<br/>Input: image:1.0.0<br/>Output: test-results<br/>Tester Signature"] J -->|Check| K["Verify Tests<br/>Passed"] K -->|Yes| L["Step 4: Sign"] L -->|Record| M["Sign Link<br/>Input: image<br/>Output: signature<br/>Signer Signature"] M -->|Bundle| N["Layout<br/>Manifest<br/>Required Steps<br/>Authorizations"] N -->|Contains| O["Code-review<br/>Step"] N -->|Contains| P["Build<br/>Step"] N -->|Contains| Q["Test<br/>Step"] N -->|Contains| R["Sign<br/>Step"] O -->|Point to| C P -->|Point to| G Q -->|Point to| J R -->|Point to| M N -->|Deliver| S["In-Toto<br/>Metadata<br/>Links + Layout"] S -->|Verify| T["Verifier<br/>Checks"] T -->|Verify| U["All Steps<br/>Executed?"] T -->|Verify| V["All Signatures<br/>Valid?"] T -->|Verify| W["No Tampering<br/>Detected?"] U -->|Pass| X["Integrity<br/>Verified"] V -->|Pass| X W -->|Pass| X U -->|Fail| Y["Rejected<br/>Step Missing"] V -->|Fail| Y W -->|Fail| Y X -->|Result| Z["Image Safe<br/>to Deploy<br/>Provenance Complete"] Y -->|Result| AA["Reject Image<br/>Do Not Deploy"] style A fill:#ccffcc style C fill:#99ccff style G fill:#99ccff style J fill:#99ccff style M fill:#99ccff style N fill:#99ccff style S fill:#ffff99 style Z fill:#99ff99Link Metadata
A link is a signed document recording what step was executed (e.g., "build", "test", "review"), who executed it, when it happened, and what changed as inputs versus outputs (documented by hashes).
{ "_type": "link", "name": "build", "materials": { "src/main.py": { "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" }, "Dockerfile": { "sha256": "5f6c3a7e2e8c1f9d4a3b2c1e0f9d8c7b6a5e4d3c2b1a0f9e8d7c6b5a4e3d2c" } }, "products": { "myapp:1.0.0": { "sha256": "abc123def456789fedcba987654321abc123def456789fedcba987654321abc" } }, "byproducts": { "return-value": 0, "stdout": "Build completed successfully", "stderr": "" }, "environment": { "golang-version": "1.22", "os": "Linux", "docker-version": "24.0.5" }, "signature": { "keyid": "builder-key-abc123", "sig": "abcd1234efgh5678..." }}Layout
A layout is a manifest specifying required supply chain steps and authorization:
{ "_type": "layout", "steps": [ { "name": "code-review", "expected_materials": [["src/**"]], "expected_products": [["src/**"]], "pubkeys": ["reviewer-key-id"], "threshold": 1, "expected_command": ["review"] }, { "name": "build", "expected_materials": [["src/**"]], "expected_products": [["*"]], "pubkeys": ["builder-key-id"], "threshold": 1, "expected_command": ["./build.sh"] }, { "name": "test", "expected_materials": [["*"]], "expected_products": [["test-results.json"]], "pubkeys": ["tester-key-id"], "threshold": 1, "expected_command": ["pytest"] } ], "inspect": [], "keys": { "reviewer-key-id": {"keyid_hash_algorithms": ["sha256"]}, "builder-key-id": {"keyid_hash_algorithms": ["sha256"]}, "tester-key-id": {"keyid_hash_algorithms": ["sha256"]} }}CleanStart In-Toto Integration
Generating In-Toto Links
CleanStart automatically generates In-Toto links for each build step:
# Build with In-Toto link generationcleanimg-init --build \ --image myapp:1.0.0 \ --in-toto-layout layout.json \ --generate-links # Output files:# - build.link (signed link for build step)# - sbom.link (signed link for SBOM generation)# - scan.link (signed link for vulnerability scanning)Multi-Step Workflow
# .github/workflows/secure-build.ymlname: Secure Build with In-Toto on: [push] jobs: code-review: runs-on: ubuntu-latest permissions: id-token: write steps: - uses: actions/checkout@v4 - name: Review code run: | # Simulate code review echo "Code review passed" > review-report.txt - name: Generate In-Toto link run: | in-toto-run \ --key-path review-key \ --links-dir links/ \ --materials "." \ --products "review-report.txt" \ -- echo "code-review completed" build: needs: code-review runs-on: ubuntu-latest permissions: id-token: write steps: - uses: actions/checkout@v4 - name: Build image run: | cleanimg-init --build --image myapp:1.0.0 - name: Generate build link run: | in-toto-run \ --key-path builder-key \ --links-dir links/ \ --materials "src/" \ --products "myapp.tar" \ -- cleanimg-init --build --image myapp:1.0.0 test: needs: build runs-on: ubuntu-latest permissions: id-token: write steps: - uses: actions/checkout@v4 - name: Run tests run: pytest tests/ - name: Generate test link run: | in-toto-run \ --key-path tester-key \ --links-dir links/ \ --materials "src/,tests/" \ --products "test-results.json" \ -- pytest tests/ --json test-results.jsonLayout Verification
Verify Supply Chain Completion
# Verify all steps completed and links are validin-toto-verify \ --layout layout.json \ --links-dir links/ \ --key reviewer.pub builder.pub tester.pub # Output:# Step 'code-review' verified (link signed by reviewer)# Step 'build' verified (link signed by builder)# Step 'test' verified (link signed by tester)# All materials and products accounted for# No modifications detected# Supply chain integrity verifiedVerify Artifact Integrity
The layout verification ensures complete supply chain integrity by checking multiple aspects. First, all steps specified in the layout must have corresponding link files. Second, each link must be cryptographically signed by a key authorized for that step. Third, the materials (inputs) for each step must match the products (outputs) of the previous step, creating an unbroken chain. Fourth, artifact hashes remain unchanged between steps, making tampering detectable. Finally, the threshold of required signatures for each step must be met.
The verification process follows these steps: Code review is signed by the reviewer, after which files remain unchanged. The build is signed by the builder, and artifacts remain unchanged. Testing is signed by the tester, and results remain unchanged. Once all these steps are verified, the complete supply chain is confirmed as verified.
Creating In-Toto Layouts
Basic Layout
{ "_type": "layout", "version": 1, "expires": "2026-12-31T23:59:59Z", "keys": {}, "steps": [ { "name": "build", "expected_command": ["./build.sh"], "pubkeys": ["builder-key-abc123"], "threshold": 1, "expected_materials": [ ["src/**"] ], "expected_products": [ ["*.tar.gz"] ] } ], "inspect": []}Complex Layout with Multiple Reviewers
{ "_type": "layout", "steps": [ { "name": "code-review", "expected_command": ["github-review"], "pubkeys": [ "alice@company.com", "bob@company.com" ], "threshold": 2, "expected_materials": [["src/**"]], "expected_products": [["src/**"]] }, { "name": "security-scan", "expected_command": ["grype", "--json"], "pubkeys": ["security-scanner@company.com"], "threshold": 1, "expected_materials": [["*.tar.gz"]], "expected_products": [["security-scan.json"]] }, { "name": "build", "expected_command": ["./build.sh"], "pubkeys": ["ci-builder@company.com"], "threshold": 1, "expected_materials": [["src/**"], ["security-scan.json"]], "expected_products": [["myapp:1.0.0"]] }, { "name": "release-approval", "expected_command": ["approval"], "pubkeys": [ "release-lead@company.com" ], "threshold": 1, "expected_materials": [["myapp:1.0.0"]], "expected_products": [["APPROVED"]] } ]}In-Toto with Hardware Security Modules
Signing Links with HSM
To sign links using a hardware security module, configure the PKCS11 module settings first. Then generate links using the HSM key instead of a regular file-based key. The key material never leaves the hardware, providing maximum security.
# Configure HSMexport PKCS11_MODULE="/usr/lib/softhsm/libsofthsm2.so"export PKCS11_SLOT="0"export PKCS11_PIN="1234" # Generate link using HSM keyin-toto-run \ --key-path hsm:builder-key-id \ --links-dir links/ \ --materials "src/" \ --products "build-output/" \ -- ./build.sh # Link is cryptographically signed by HSM (key never leaves hardware)Verifying HSM Signatures
When verifying signatures from HSM keys, provide the HSM key references instead of file paths. The verification process validates the cryptographic signatures using the public key components.
# Verify signatures from HSM keysin-toto-verify \ --layout layout.json \ --links-dir links/ \ --key hsm:builder-key.pub hsm:tester-key.pub # Verification includes HSM signature validationIntegration with Kubernetes
In-Toto Verification Admission Controller
apiVersion: v1kind: ConfigMapmetadata: name: in-toto-layout namespace: defaultdata: layout.json: | { "_type": "layout", "steps": [ { "name": "build", "pubkeys": ["builder-key"], "threshold": 1 } ] } ---apiVersion: admissionregistration.k8s.io/v1kind: ValidatingWebhookConfigurationmetadata: name: in-toto-verificationwebhooks:- name: verify-in-toto.company.com failurePolicy: Fail sideEffects: None admissionReviewVersions: ["v1"] clientConfig: service: name: in-toto-verifier namespace: default path: "/verify" rules: - operations: ["CREATE"] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"]Deployment verification:
# Before deploying, verify In-Toto layout compliancein-toto-verify \ --layout layout.json \ --links-dir ci/links/ \ --key builder.pub security.pub # If verification passes, Kubernetes admission controller allows pod# If verification fails, pod is rejectedCompliance Benefits
Supply Chain Security
In-Toto provides evidence for compliance frameworks including NIST 800-53 SA-3 (development life cycle controls), FedRAMP SI-2 (flaw identification and remediation), CISA SSDF PS2 (secure development practices), and ISO 27001 A.14.2 (secure development policy).
# Generate compliance report with In-Toto evidencecleanimg-init --image myapp:1.0.0 \ --compliance fedramp \ --in-toto-layout layout.json \ --links-dir links/ \ --output compliance-report.json # Report includes:# - All supply chain steps verified# - Signatures from authorized parties# - Artifact integrity guarantees# - Timestamp evidenceAudit Trail
Every In-Toto link creates permanent audit evidence of what happened, who performed the action, and when:
# Extract audit evidence from linksfor link in links/*.link; do echo "=== $(basename $link) ===" jq '{step: .name, signer: .signature.keyid, timestamp: .time}' "$link"done # Output shows complete audit trail:# === code-review.link ===# {# "step": "code-review",# "signer": "alice@company.com",# "timestamp": "2025-10-04T14:10:00Z"# }Troubleshooting In-Toto
Link Verification Fails
When link verification fails, check the signatures on each link file. Run verification with verbose output to see detailed error messages. Common issues include keys that don't match, input files that changed, output files that changed, or missing link files for required steps.
# Check link signaturesin-toto-verify \ --layout layout.json \ --links-dir links/ \ --key builder.pub \ --verbose # Common issues:# 1. Key mismatch: Link signed by different key than expected# 2. Material mismatch: Input files changed# 3. Product mismatch: Output files changed# 4. Missing link: Step not executedMaterial Tracking Issues
When material tracking has issues, use the in-toto-record command to manually track materials and products. Start recording before the build step, run the build, then stop recording to generate a link with detailed material tracking.
# Debug material hashingin-toto-record \ --key builder-key \ --links-dir links/ \ start build # Run build step./build.sh in-toto-record \ --key builder-key \ --links-dir links/ \ stop build # Generates build.link with detailed material trackingBest Practices
Include All Supply Chain Steps
Layouts should include every decision point that affects the final artifact. This means including code review for human oversight, security scanning for vulnerability detection, build for artifact creation, testing for quality assurance, and release approval for final authorization.
Use Multi-Signer Layouts
Require multiple authorized parties for critical steps to prevent single points of compromise:
{ "steps": [ { "name": "release-approval", "pubkeys": [ "release-manager@company.com", "security-lead@company.com" ], "threshold": 2 } ]}Regular Layout Rotation
Update layout expiration and rotate keys periodically to maintain security:
# Annual key rotationin-toto-keygen -t rsa -n builder-2025 # Update layout with new keyjq '.keys.builder = new_key_content' layout.json > layout-2025.jsonSee Also
Provenance Chaining: provenance-chaining.md — Cross-stage verification. SLSA Level 4: slsa-level-4.md — Build provenance details. Image Signing: image-signing-sigstore.md — Cryptographic signing.