Knowledge Hub

In-Toto Attestation: Supply Chain Link Verification

Generating in-toto Attestations for Build Steps

In-Toto is a framework for securing the integrity of the software supply chain by creating verifiable links between software artifacts and the steps that produced them. Each step in the supply chain (code review, build, test, sign) produces a "link" document that records what was done and what changed.

CleanStart uses In-Toto to create verifiable supply chain links, enabling downstream verification that artifacts went through all required steps and were not tampered with.

In-Toto Concepts

The following diagram illustrates the in-toto supply chain link verification flow:

graph TB    A["Source Code<br/>Commit"] -->|Step 1| B["Code Review<br/>Link"]     B -->|Signed| C["Review Link<br/>Input: src/*<br/>Output: approved<br/>Reviewer Signature"]     C -->|Required| D{All Steps<br/>Complete?}     D -->|No| E["Review Link<br/>Locked<br/>Pending"]     D -->|Yes| F["Step 2: Build"]     F -->|Record| G["Build Link<br/>Input: src/* + Dockerfile<br/>Output: image:1.0.0<br/>Builder Signature"]     G -->|Check| H["Verify Build<br/>Happened"]     H -->|Yes| I["Step 3: Test"]     I -->|Record| J["Test Link<br/>Input: image:1.0.0<br/>Output: test-results<br/>Tester Signature"]     J -->|Check| K["Verify Tests<br/>Passed"]     K -->|Yes| L["Step 4: Sign"]     L -->|Record| M["Sign Link<br/>Input: image<br/>Output: signature<br/>Signer Signature"]     M -->|Bundle| N["Layout<br/>Manifest<br/>Required Steps<br/>Authorizations"]     N -->|Contains| O["Code-review<br/>Step"]    N -->|Contains| P["Build<br/>Step"]    N -->|Contains| Q["Test<br/>Step"]    N -->|Contains| R["Sign<br/>Step"]     O -->|Point to| C    P -->|Point to| G    Q -->|Point to| J    R -->|Point to| M     N -->|Deliver| S["In-Toto<br/>Metadata<br/>Links + Layout"]     S -->|Verify| T["Verifier<br/>Checks"]     T -->|Verify| U["All Steps<br/>Executed?"]    T -->|Verify| V["All Signatures<br/>Valid?"]    T -->|Verify| W["No Tampering<br/>Detected?"]     U -->|Pass| X["Integrity<br/>Verified"]    V -->|Pass| X    W -->|Pass| X     U -->|Fail| Y["Rejected<br/>Step Missing"]    V -->|Fail| Y    W -->|Fail| Y     X -->|Result| Z["Image Safe<br/>to Deploy<br/>Provenance Complete"]     Y -->|Result| AA["Reject Image<br/>Do Not Deploy"]     style A fill:#ccffcc    style C fill:#99ccff    style G fill:#99ccff    style J fill:#99ccff    style M fill:#99ccff    style N fill:#99ccff    style S fill:#ffff99    style Z fill:#99ff99

Link Metadata

A link is a signed document recording what step was executed (e.g., "build", "test", "review"), who executed it, when it happened, and what changed as inputs versus outputs (documented by hashes).

{  "_type": "link",  "name": "build",  "materials": {    "src/main.py": {      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"    },    "Dockerfile": {      "sha256": "5f6c3a7e2e8c1f9d4a3b2c1e0f9d8c7b6a5e4d3c2b1a0f9e8d7c6b5a4e3d2c"    }  },  "products": {    "myapp:1.0.0": {      "sha256": "abc123def456789fedcba987654321abc123def456789fedcba987654321abc"    }  },  "byproducts": {    "return-value": 0,    "stdout": "Build completed successfully",    "stderr": ""  },  "environment": {    "golang-version": "1.22",    "os": "Linux",    "docker-version": "24.0.5"  },  "signature": {    "keyid": "builder-key-abc123",    "sig": "abcd1234efgh5678..."  }}

Layout

A layout is a manifest specifying required supply chain steps and authorization:

{  "_type": "layout",  "steps": [    {      "name": "code-review",      "expected_materials": [["src/**"]],      "expected_products": [["src/**"]],      "pubkeys": ["reviewer-key-id"],      "threshold": 1,      "expected_command": ["review"]    },    {      "name": "build",      "expected_materials": [["src/**"]],      "expected_products": [["*"]],      "pubkeys": ["builder-key-id"],      "threshold": 1,      "expected_command": ["./build.sh"]    },    {      "name": "test",      "expected_materials": [["*"]],      "expected_products": [["test-results.json"]],      "pubkeys": ["tester-key-id"],      "threshold": 1,      "expected_command": ["pytest"]    }  ],  "inspect": [],  "keys": {    "reviewer-key-id": {"keyid_hash_algorithms": ["sha256"]},    "builder-key-id": {"keyid_hash_algorithms": ["sha256"]},    "tester-key-id": {"keyid_hash_algorithms": ["sha256"]}  }}

CleanStart In-Toto Integration

Generating In-Toto Links

CleanStart automatically generates In-Toto links for each build step:

# Build with In-Toto link generationcleanimg-init --build \  --image myapp:1.0.0 \  --in-toto-layout layout.json \  --generate-links # Output files:# - build.link (signed link for build step)# - sbom.link (signed link for SBOM generation)# - scan.link (signed link for vulnerability scanning)

Multi-Step Workflow

# .github/workflows/secure-build.ymlname: Secure Build with In-Toto on: [push] jobs:  code-review:    runs-on: ubuntu-latest    permissions:      id-token: write    steps:      - uses: actions/checkout@v4       - name: Review code        run: |          # Simulate code review          echo "Code review passed" > review-report.txt       - name: Generate In-Toto link        run: |          in-toto-run \            --key-path review-key \            --links-dir links/ \            --materials "." \            --products "review-report.txt" \            -- echo "code-review completed"   build:    needs: code-review    runs-on: ubuntu-latest    permissions:      id-token: write    steps:      - uses: actions/checkout@v4       - name: Build image        run: |          cleanimg-init --build --image myapp:1.0.0       - name: Generate build link        run: |          in-toto-run \            --key-path builder-key \            --links-dir links/ \            --materials "src/" \            --products "myapp.tar" \            -- cleanimg-init --build --image myapp:1.0.0   test:    needs: build    runs-on: ubuntu-latest    permissions:      id-token: write    steps:      - uses: actions/checkout@v4       - name: Run tests        run: pytest tests/       - name: Generate test link        run: |          in-toto-run \            --key-path tester-key \            --links-dir links/ \            --materials "src/,tests/" \            --products "test-results.json" \            -- pytest tests/ --json test-results.json

Layout Verification

Verify Supply Chain Completion

# Verify all steps completed and links are validin-toto-verify \  --layout layout.json \  --links-dir links/ \  --key reviewer.pub builder.pub tester.pub # Output:# Step 'code-review' verified (link signed by reviewer)# Step 'build' verified (link signed by builder)# Step 'test' verified (link signed by tester)# All materials and products accounted for# No modifications detected# Supply chain integrity verified

Verify Artifact Integrity

The layout verification ensures complete supply chain integrity by checking multiple aspects. First, all steps specified in the layout must have corresponding link files. Second, each link must be cryptographically signed by a key authorized for that step. Third, the materials (inputs) for each step must match the products (outputs) of the previous step, creating an unbroken chain. Fourth, artifact hashes remain unchanged between steps, making tampering detectable. Finally, the threshold of required signatures for each step must be met.

The verification process follows these steps: Code review is signed by the reviewer, after which files remain unchanged. The build is signed by the builder, and artifacts remain unchanged. Testing is signed by the tester, and results remain unchanged. Once all these steps are verified, the complete supply chain is confirmed as verified.

Creating In-Toto Layouts

Basic Layout

{  "_type": "layout",  "version": 1,  "expires": "2026-12-31T23:59:59Z",  "keys": {},  "steps": [    {      "name": "build",      "expected_command": ["./build.sh"],      "pubkeys": ["builder-key-abc123"],      "threshold": 1,      "expected_materials": [        ["src/**"]      ],      "expected_products": [        ["*.tar.gz"]      ]    }  ],  "inspect": []}

Complex Layout with Multiple Reviewers

{  "_type": "layout",  "steps": [    {      "name": "code-review",      "expected_command": ["github-review"],      "pubkeys": [        "alice@company.com",        "bob@company.com"      ],      "threshold": 2,      "expected_materials": [["src/**"]],      "expected_products": [["src/**"]]    },    {      "name": "security-scan",      "expected_command": ["grype", "--json"],      "pubkeys": ["security-scanner@company.com"],      "threshold": 1,      "expected_materials": [["*.tar.gz"]],      "expected_products": [["security-scan.json"]]    },    {      "name": "build",      "expected_command": ["./build.sh"],      "pubkeys": ["ci-builder@company.com"],      "threshold": 1,      "expected_materials": [["src/**"], ["security-scan.json"]],      "expected_products": [["myapp:1.0.0"]]    },    {      "name": "release-approval",      "expected_command": ["approval"],      "pubkeys": [        "release-lead@company.com"      ],      "threshold": 1,      "expected_materials": [["myapp:1.0.0"]],      "expected_products": [["APPROVED"]]    }  ]}

In-Toto with Hardware Security Modules

Signing Links with HSM

To sign links using a hardware security module, configure the PKCS11 module settings first. Then generate links using the HSM key instead of a regular file-based key. The key material never leaves the hardware, providing maximum security.

# Configure HSMexport PKCS11_MODULE="/usr/lib/softhsm/libsofthsm2.so"export PKCS11_SLOT="0"export PKCS11_PIN="1234" # Generate link using HSM keyin-toto-run \  --key-path hsm:builder-key-id \  --links-dir links/ \  --materials "src/" \  --products "build-output/" \  -- ./build.sh # Link is cryptographically signed by HSM (key never leaves hardware)

Verifying HSM Signatures

When verifying signatures from HSM keys, provide the HSM key references instead of file paths. The verification process validates the cryptographic signatures using the public key components.

# Verify signatures from HSM keysin-toto-verify \  --layout layout.json \  --links-dir links/ \  --key hsm:builder-key.pub hsm:tester-key.pub # Verification includes HSM signature validation

Integration with Kubernetes

In-Toto Verification Admission Controller

apiVersion: v1kind: ConfigMapmetadata:  name: in-toto-layout  namespace: defaultdata:  layout.json: |    {      "_type": "layout",      "steps": [        {          "name": "build",          "pubkeys": ["builder-key"],          "threshold": 1        }      ]    } ---apiVersion: admissionregistration.k8s.io/v1kind: ValidatingWebhookConfigurationmetadata:  name: in-toto-verificationwebhooks:- name: verify-in-toto.company.com  failurePolicy: Fail  sideEffects: None  admissionReviewVersions: ["v1"]  clientConfig:    service:      name: in-toto-verifier      namespace: default      path: "/verify"  rules:  - operations: ["CREATE"]    apiGroups: [""]    apiVersions: ["v1"]    resources: ["pods"]

Deployment verification:

# Before deploying, verify In-Toto layout compliancein-toto-verify \  --layout layout.json \  --links-dir ci/links/ \  --key builder.pub security.pub # If verification passes, Kubernetes admission controller allows pod# If verification fails, pod is rejected

Compliance Benefits

Supply Chain Security

In-Toto provides evidence for compliance frameworks including NIST 800-53 SA-3 (development life cycle controls), FedRAMP SI-2 (flaw identification and remediation), CISA SSDF PS2 (secure development practices), and ISO 27001 A.14.2 (secure development policy).

# Generate compliance report with In-Toto evidencecleanimg-init --image myapp:1.0.0 \  --compliance fedramp \  --in-toto-layout layout.json \  --links-dir links/ \  --output compliance-report.json # Report includes:# - All supply chain steps verified# - Signatures from authorized parties# - Artifact integrity guarantees# - Timestamp evidence

Audit Trail

Every In-Toto link creates permanent audit evidence of what happened, who performed the action, and when:

# Extract audit evidence from linksfor link in links/*.link; do  echo "=== $(basename $link) ==="  jq '{step: .name, signer: .signature.keyid, timestamp: .time}' "$link"done # Output shows complete audit trail:# === code-review.link ===# {#   "step": "code-review",#   "signer": "alice@company.com",#   "timestamp": "2025-10-04T14:10:00Z"# }

Troubleshooting In-Toto

Link Verification Fails

When link verification fails, check the signatures on each link file. Run verification with verbose output to see detailed error messages. Common issues include keys that don't match, input files that changed, output files that changed, or missing link files for required steps.

# Check link signaturesin-toto-verify \  --layout layout.json \  --links-dir links/ \  --key builder.pub \  --verbose # Common issues:# 1. Key mismatch: Link signed by different key than expected# 2. Material mismatch: Input files changed# 3. Product mismatch: Output files changed# 4. Missing link: Step not executed

Material Tracking Issues

When material tracking has issues, use the in-toto-record command to manually track materials and products. Start recording before the build step, run the build, then stop recording to generate a link with detailed material tracking.

# Debug material hashingin-toto-record \  --key builder-key \  --links-dir links/ \  start build # Run build step./build.sh in-toto-record \  --key builder-key \  --links-dir links/ \  stop build # Generates build.link with detailed material tracking

Best Practices

Include All Supply Chain Steps

Layouts should include every decision point that affects the final artifact. This means including code review for human oversight, security scanning for vulnerability detection, build for artifact creation, testing for quality assurance, and release approval for final authorization.

Use Multi-Signer Layouts

Require multiple authorized parties for critical steps to prevent single points of compromise:

{  "steps": [    {      "name": "release-approval",      "pubkeys": [        "release-manager@company.com",        "security-lead@company.com"      ],      "threshold": 2    }  ]}

Regular Layout Rotation

Update layout expiration and rotate keys periodically to maintain security:

# Annual key rotationin-toto-keygen -t rsa -n builder-2025 # Update layout with new keyjq '.keys.builder = new_key_content' layout.json > layout-2025.json