Linking Provenance Across Multi-Stage Build Pipelines
Provenance chaining creates an unbroken cryptographic link from source code through compilation, packaging, containerization, scanning, signing, and deployment. Each stage produces attestations that reference previous stages, creating a verifiable chain of custody.
CleanStart implements full provenance chaining, enabling security teams to prove the exact lineage of any deployed artifact: which source code was built, by which builders, using which dependencies, scanned with which tools, and approved by which policies.
The Provenance Chain
Complete Flow
The provenance chain begins with source code in git, documented with a commit attestation. The build environment (Cloud Build) generates build provenance using SLSA standards. The resulting build artifact (container image) is documented with an SBOM attestation. SBOM analysis (Syft) produces vulnerability attestations. Vulnerability scanning (Grype) generates VEX attestations. VEX generation (CleanStart) produces policy evaluation attestations. Policy evaluation (OPA) creates approval attestations. Security review (human) generates deployment attestations. Finally, the package is deployed to the Kubernetes cluster. Each stage links to the previous through cryptographic attestations, creating an unbroken chain of evidence.
Stage 1: Source Code Attestation
Git Commit Signing
# Configure git to sign all commitsgit config --global user.signingKey my-gpg-key-idgit config --global commit.gpgSign true # Create signed commitgit commit -m "Release v1.0.0"# Output: [main abc123d] Release v1.0.0# gpg: Good signature from "Developer Name" # Verify signature in provenancegit log --show-signature# gpg: Good signature from "Developer Name <dev@company.com>"# Signature metadata becomes part of provenance chainSource Code Attestation Format
{ "type": "attestation/source", "subject": { "name": "myapp", "uri": "https://github.com/company/myapp", "digest": { "sha256": "abc123def456..." } }, "predicateType": "https://slsa.dev/sourceverify/v1", "predicate": { "commit": "abc123def456", "author": { "name": "Alice Developer", "email": "alice@company.com" }, "message": "Release v1.0.0", "timestamp": "2025-10-04T14:00:00Z", "gpgSignature": { "keyId": "ABCD1234EFGH5678", "verified": true } }}Stage 2: Build Provenance Attestation
SLSA Provenance (Level 4)
Build provenance references source attestation:
{ "type": "attestation/slsa", "predicateType": "https://slsa.dev/provenance/v0.2", "predicate": { "buildDefinition": { "externalParameters": { "source": { "uri": "https://github.com/company/myapp", "digest": { "sha256": "abc123def456..." // References source attestation }, "entrypoint": "Dockerfile" } }, "resolvedDependencies": [ { "uri": "npm://typescript@5.1.6", "digest": { "npm": "sha512:typescript-hash..." } } ] }, "runDetails": { "builder": { "id": "https://cloudbuild.googleapis.com/projects/apk-test-442304/builds/build-id-123" }, "byproducts": [ { "uri": "gcr.io/apk-test-442304/myapp:1.0.0@sha256:image-hash", "digest": { "sha256": "image-hash..." } } ] } }}CleanStart generates SLSA Level 4 provenance automatically:
# Build produces signed provenancecleanimg-init --build --image myapp:1.0.0 --slsa-level 4 # Provenance stored with imagecosign attach attestation \ gcr.io/apk-test-442304/myapp:1.0.0 \ --attestation provenance.json \ --type slsaprovenanceStage 3: SBOM Attestation
SBOM with Provenance Reference
{ "type": "attestation/sbom", "predicateType": "https://spdx.dev/predicate/spdx/v0.2", "subject": { "name": "gcr.io/apk-test-442304/myapp:1.0.0", "digest": { "sha256": "image-hash-from-build-provenance" } }, "predicate": { "spdxVersion": "SPDX-3.0.0", "provenanceReference": { "buildId": "https://cloudbuild.googleapis.com/projects/apk-test-442304/builds/build-id-123" }, "packages": [ { "SPDXID": "SPDXRef-typescript", "name": "typescript", "version": "5.1.6", "downloadLocation": "npm://typescript@5.1.6", "checksums": [ { "algorithm": "SHA512", "checksumValue": "typescript-hash..." } ] } ] }}Generate SBOM with provenance reference:
# Generate SBOM with build provenance linkcleanimg-init --image myapp:1.0.0 \ --sbom app.spdx \ --reference-provenance # Attach SBOM to imagecosign attach sbom gcr.io/apk-test-442304/myapp:1.0.0 \ --sbom app.spdx \ --type spdxStage 4: Vulnerability Attestation
Scanning Attestation with SBOM Reference
{ "type": "attestation/vulnerability", "predicateType": "https://grype.dev/predicate/vulnerability-scan/v1", "subject": { "name": "gcr.io/apk-test-442304/myapp:1.0.0", "digest": { "sha256": "image-hash..." } }, "predicate": { "sbomReference": "sha256:sbom-hash-from-sbom-attestation", "scanner": { "name": "grype", "version": "0.68.1" }, "scanTime": "2025-10-04T14:15:00Z", "vulnerabilities": [ { "cveId": "CVE-2024-5678", "severity": "MODERATE", "packageName": "openssl", "packageVersion": "3.1.4", "affectedRange": "<3.1.5" } ] }}Scan with explicit provenance linking:
# Scan image, reference SBOM attestationgrype \ --image gcr.io/apk-test-442304/myapp:1.0.0 \ --sbom-version spdx-3.0 \ --output json > vulnerabilities.json # Attach vulnerability attestationcosign attach attestation \ gcr.io/apk-test-442304/myapp:1.0.0 \ --attestation vulnerabilities.json \ --type vulnattestv1Stage 5: VEX Attestation
VEX with Vulnerability Reference
{ "type": "attestation/vex", "predicateType": "https://openvex.dev/predicate/vex/v1", "subject": { "name": "gcr.io/apk-test-442304/myapp:1.0.0" }, "predicate": { "vulnerabilityAttestationReference": "sha256:vuln-attestation-hash", "sbomReference": "sha256:sbom-hash", "statements": [ { "vulnerability": { "name": "CVE-2024-5678" }, "status": "not_affected", "justification": "vulnerable_code_not_in_execute_path", "impact_statement": "OpenSSL vulnerability in certificate parsing; application never parses certificates", "evidence": { "source": "static_analysis", "confidence": 0.98, "reference_sbom": "sha256:sbom-hash" } } ] }}Stage 6: Policy Evaluation Attestation
OPA Policy Check
{ "type": "attestation/policy-evaluation", "predicateType": "https://openpolicyagent.org/predicate/policy-eval/v1", "subject": { "name": "gcr.io/apk-test-442304/myapp:1.0.0" }, "predicate": { "evaluationTime": "2025-10-04T14:20:00Z", "policies": [ { "name": "require-slsa-provenance", "result": "PASS", "reason": "Image has SLSA Level 4 provenance attestation" }, { "name": "no-high-severity-vulnerabilities", "result": "PASS", "reason": "Scan shows 0 high-severity vulnerabilities" }, { "name": "all-vex-reviewed", "result": "PASS", "reason": "All affected vulnerabilities have VEX status" }, { "name": "approved-base-images", "result": "PASS", "reason": "Base image alpine:3.19 is in approved registry" } ], "overallResult": "PASS" }}Configure policy evaluation:
# opa-policies.regopackage image_security import data.attestations.slsaimport data.attestations.sbomimport data.attestations.vex # Check SLSA Level 4slsa_compliant { slsa.level >= 4} # Check vulnerability statusvex_compliant { count(vex.statements[{status: "affected"}]) == 0} # Overall pass/failallow { slsa_compliant vex_compliant}Apply policy and generate attestation:
# Evaluate policiesconftest test \ --policy opa-policies.rego \ --input attestations.json # Generate policy evaluation attestationcosign attest \ --predicate policy-eval.json \ gcr.io/apk-test-442304/myapp:1.0.0Stage 7: Approval Attestation
Human Approval Signature
{ "type": "attestation/approval", "predicateType": "https://codenotary.dev/predicate/approval/v1", "subject": { "name": "gcr.io/apk-test-442304/myapp:1.0.0", "digest": { "sha256": "image-hash..." } }, "predicate": { "approver": { "name": "Security Lead", "email": "security@company.com", "title": "Director of Security" }, "approvalTime": "2025-10-04T14:25:00Z", "referencedAttestations": { "slsa_provenance": "sha256:provenance-hash", "sbom": "sha256:sbom-hash", "vex": "sha256:vex-hash", "vulnerability_scan": "sha256:scan-hash", "policy_evaluation": "sha256:policy-hash" }, "approvalReason": "All policies passed, no unresolved vulnerabilities, provenance verified", "approvalScope": "production-deployment" }}Stage 8: Deployment Attestation
Kubernetes Attestation
{ "type": "attestation/deployment", "predicateType": "https://kubernetes.io/predicate/deployment/v1", "subject": { "name": "gcr.io/apk-test-442304/myapp:1.0.0" }, "predicate": { "deploymentTime": "2025-10-04T14:30:00Z", "environment": "production", "cluster": "prod-us-central1-1", "namespace": "default", "deployment": "myapp", "approvalChain": [ { "type": "approval", "signer": "security@company.com", "timestamp": "2025-10-04T14:25:00Z" }, { "type": "policy-evaluation", "tool": "conftest", "timestamp": "2025-10-04T14:20:00Z", "result": "pass" } ], "imageAttestations": { "slsa": "sha256:provenance-hash", "sbom": "sha256:sbom-hash", "vulnerabilities": "sha256:scan-hash", "vex": "sha256:vex-hash" } }}Deploy with attestation verification:
# Admission controller verifies entire provenance chainkubectl apply -f deployment.yaml # Controller checks:# 1. Image has SLSA Level 4 provenance# 2. Provenance source matches approved repositories# 3. All policies passed# 4. Approval attestation present and valid# 5. All attestations in immutable registry (Rekor) # If any check fails: reject deploymentCross-Stage Verification
Verifying the Complete Chain
# Get image referenceIMAGE="gcr.io/apk-test-442304/myapp:1.0.0" # Verify each stageecho "1. Checking SLSA provenance..."cosign verify-attestation $IMAGE --type slsaprovenance echo "2. Checking SBOM..."cosign verify-sbom $IMAGE echo "3. Checking vulnerability scan..."cosign verify-attestation $IMAGE --type vulnattestv1 echo "4. Checking VEX..."cosign verify-attestation $IMAGE --type vex echo "5. Checking policy evaluation..."cosign verify-attestation $IMAGE --type slsa-policy echo "6. Checking approval..."cosign verify-attestation $IMAGE --type approval # Extract full chain for auditcosign verify-attestation $IMAGE --type slsaprovenance | jq -r '.payload' | base64 -d > provenance.json# ... repeat for each attestation typeBuilding Provenance-Aware Applications
// go/verify-provenance.gopackage main import ( "github.com/sigstore/cosign/cmd/cosign/cli/verify") func verifyDeployment(image string) error { // Verify entire provenance chain attestations, err := verify.VerifyAttestations(image, []string{ "slsaprovenance", "spdx", "vulnattestv1", "vex", "approval", }) if err != nil { return err } // Check attestation references form valid chain for i := 0; i < len(attestations)-1; i++ { current := attestations[i] next := attestations[i+1] // Verify next references current if !referencesAttestation(next, current) { return errors.New("provenance chain broken at stage " + string(i)) } } return nil}Compliance with Provenance Chains
FedRAMP Supply Chain Security
Provenance chains satisfy FedRAMP requirements by implementing SA-3, which requires tracking system development with linked attestations throughout the development lifecycle. Additionally, SA-4 documents acquisition with reference chains to verify supply chain integrity. Finally, SI-4 verifies system monitoring through deployment attestation to ensure continuous oversight.
CISA SSDF Practices
Provenance chains support PS2.1 by protecting against unauthorized changes through signing at each stage, with cryptographic evidence of who made changes and when. PS2.2 ensures artifact integrity through hash verification throughout the chain, making tampering detectable.
Troubleshooting Chain Verification
Missing Attestation
If a stage in the chain is missing, identify which attestations are absent and regenerate them:
# Identify missing attestationscosign verify-attestation $IMAGE --type slsaprovenance || echo "MISSING: SLSA"cosign verify-sbom $IMAGE || echo "MISSING: SBOM"cosign verify-attestation $IMAGE --type vulnattestv1 || echo "MISSING: Vulnerability" # Re-scan and attachgrype $IMAGE --output json | cosign attach attestation \ --attestation - --type vulnattestv1 $IMAGEChain Verification Fails
When verification fails, run detailed verification with error output to see what's wrong. Check the certificate chain to ensure all keys are valid and properly configured.
# Detailed verification with error outputcosign verify-attestation $IMAGE --type slsaprovenance --verbose # Check certificate chaincosign verify-attestation $IMAGE --certificate-identity-regexp ".*" \ --certificate-oidc-issuer "https://token.actions.githubusercontent.com"See Also
SLSA Level 4: slsa-level-4.md — Hermetic build provenance. Image Signing: image-signing-sigstore.md — Cryptographic signatures. In-Toto Attestation: in-toto-attestation.md — Artifact verification framework.