Customizing CleanStart Images Through CI/CD: From Source to Signed Artifact
Every organization needs a repeatable, auditable process for customizing CleanStart images and deploying them to production. Manual customization breaks reproducibility, auditability, and compliance — the pipeline ensures all customization is version-controlled, scanned, signed, and traceable to a Git commit.
Part 1: Why CI/CD Customization Matters
The Core Principle
All customization happens through the pipeline, never manually. This is the foundational principle of secure and maintainable container operations. Manual customization—ad-hoc docker commits, manual registry pushes, one-off deployments—breaks every aspect of modern software supply chain security.
Manual customization destroys reproducibility because the same source code produces different images depending on when you build, who built it, and what tools they had locally. If you rebuild the image tomorrow, you might get different results because local dependencies changed, or a base image was updated. This inconsistency makes debugging impossible and prevents reliable rollbacks. Manual customization completely breaks auditability because there's no record of who made changes, when, or why. Changes exist only in a developer's shell history and brain. Supply chain provenance is broken because the image cannot be traced back to a specific Git commit. Security scanning and signing are bypassed entirely because there's no formal pipeline to enforce them.
A proper CI/CD pipeline ensures that every image is built exclusively from version-controlled source code—Dockerfile, application code, configuration—all stored in Git with a complete history. Every build is automatically scanned for vulnerabilities before being released. Every image is cryptographically signed with a verifiable key, enabling downstream systems to reject unsigned or tampered images. Every build produces a complete Software Bill of Materials (SBOM) documenting every package, library, and component in the image. Most importantly, every build is traceable to a specific Git commit, enabling forensics and rollback.
Supply Chain Security Standard: SLSA
SLSA (Supply-chain Levels for Software Artifacts) is an industry standard framework for software supply chain security, created by Google and endorsed by the Linux Foundation. It defines four maturity levels for supply chain security, each building on the previous.
Level 1 requires source code version control. All source code—application code, Dockerfiles, configuration files—must be version-controlled in a Git repository (or equivalent). This enables auditability and rollback. Many organizations stop here, assuming version control is sufficient.
Level 2 adds automated CI/CD builds and cryptographic signing. Builds happen automatically in a CI/CD system (GitHub Actions, GitLab CI, Cloud Build), not on developer machines. The system maintains an audit log of every build. Generated artifacts (container images) are cryptographically signed, enabling verification that the artifact came from the official pipeline.
Level 3 adds provenance verification and hardened builds. The CI/CD system generates cryptographic provenance attestations proving how the image was built, what inputs were used, and what tooling was applied. Provenance is machine-readable and can be cryptographically verified. The build environment is hardened to prevent tampering.
Level 4 requires two-person review and offline signing keys. Any changes to the build process require approval from at least two authorized individuals. Signing keys are kept offline and used only during formal release processes, preventing compromise of actively-used keys.
CleanStart pipelines are designed to meet SLSA Level 3 out of the box, providing provenance verification, automated builds, cryptographic signing, and complete auditability. Organizations can extend to SLSA Level 4 by adding additional approval gates and offline key management.
Part 2: The Secure Customization Pipeline (5 Stages)
Every build goes through five sequential stages, each with specific tasks and validation gates.
Stage 1: Source (Git). The process begins when you commit application code and a Dockerfile to Git. The pipeline is triggered automatically on the commit, ensuring the build captures the exact state of the code.
Stage 2: Verify Base Image. Before customization begins, the pipeline pulls the CleanStart base image and verifies its authenticity. The signature is checked using cosign verify, and vulnerability scan results are reviewed to ensure the foundation is secure.
Stage 3: Customize & Build. The pipeline runs the build using either a multi-stage Dockerfile (for code changes) or cleanimg-customize (for static files). After the image is built, it is scanned with grype to identify vulnerabilities. Any high or critical vulnerabilities cause the build to be rejected.
Stage 4: Generate Provenance & Sign. Once the build passes scanning, the pipeline generates a Software Bill of Materials (SBOM) with syft, creates a provenance attestation for SLSA compliance, and cryptographically signs the image with cosign.
Stage 5: Push & Register. The signed image is pushed to the registry, and the SBOM is attached as an image attestation. Provenance metadata is uploaded, and the deployment system is notified that a new image is ready.
Stage 1: Source (Git)
Your Git repository has a standard structure that supports the CI/CD pipeline. At the root level, place the Dockerfile that defines your multi-stage build process. The .github/workflows/ directory contains the CI/CD pipeline configuration file (build.yml). Application code lives in the src/ directory and can include Python files like main.py and app.py. Configuration files go in the config/ directory, such as cleanimg-init.toml for initialization settings. Static assets and certificates are stored in the certs/ directory, including the corporate CA bundle used by cleanimg-customize.
Stage 2: Verify Base Image
Before customizing a CleanStart image, verify it:
# Pull base imagedocker pull cleanstart/postgresql:15-prod@sha256:abc123... # Verify signaturecosign verify --key https://keys.cleanstart.dev/cosign.pub \ cleanstart/postgresql:15-prod@sha256:abc123... # Check vulnerability scan (from image attestation)grype --show-suppressed cleanstart/postgresql:15-prod@sha256:abc123...Stage 3: Customize & Build
Run the build (Dockerfile or cleanimg-customize):
# Multi-stage builddocker build -t myregistry/app:latest -f Dockerfile . # Or use cleanimg-customize v0.3.0 (spec-based)cleanimg-customize build \ --spec postgres-spec.yaml \ --tag myregistry/postgres:15-custom # Scan the resultgrype myregistry/app:latest --fail-on high# Exit code 0 = pass, non-zero = fail (reject the build)Stage 4: Generate Provenance & Sign
Attach metadata and sign:
# Generate SBOM (Software Bill of Materials)syft myregistry/app:latest -o json > sbom.json # Create SLSA provenanceslsa-provenance generate \ --builder-image myregistry/app:latest \ --materials ./Dockerfile \ --environment CI_COMMIT_SHA=$GIT_COMMIT \ > provenance.json # Sign image with cosign (OIDC keyless on GitHub)cosign sign --key cosign.key myregistry/app:latest# Or on GitHub (OIDC):cosign sign --key ghcr.io/cosign/keys myregistry/app:latest # Attach attestationscosign attach attestation --attestation sbom.json \ myregistry/app:latestcosign attach attestation --attestation provenance.json \ myregistry/app:latestStage 5: Push & Register
Push the signed image and notify the system:
# Push to registry (with SBOM and provenance attached)docker push myregistry/app:latest # Register in image registry (optional, for governance)curl -X POST https://image-registry/api/images \ -H "Authorization: Bearer $TOKEN" \ -d @- <<EOF{ "image": "myregistry/app:latest", "git_ref": "$GIT_COMMIT", "sbom": "$(base64 sbom.json)", "signed": true, "vulnerability_scan": "passed"}EOFPart 3: GitHub Actions — Complete Workflow
This is a production-ready .github/workflows/build.yml file:
name: Build and Sign Image on: push: branches: [main, develop] paths: - 'Dockerfile' - 'src/**' - 'config/**' - '.github/workflows/build.yml' pull_request: branches: [main] env: REGISTRY: ghcr.io IMAGE_NAME: ${{ github.repository }} jobs: build: runs-on: ubuntu-latest permissions: contents: read packages: write id-token: write # For cosign OIDC steps: # 1. Setup - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 # Full history for versioning - name: Set up Docker uses: docker/setup-buildx-action@v2 # 2. Verify base image (if using CleanStart) - name: Verify CleanStart Base Image run: | docker pull cleanstart/postgresql:15-prod@sha256:abc123... # Verify signature cosign verify --key https://keys.cleanstart.dev/cosign.pub \ cleanstart/postgresql:15-prod@sha256:abc123... # Scan base image grype cleanstart/postgresql:15-prod@sha256:abc123... \ --fail-on medium || true # 3. Build image - name: Build Docker image uses: docker/build-push-action@v5 with: context: . push: false load: true tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} outputs: type=docker # 4. Scan for vulnerabilities - name: Scan image for vulnerabilities run: | # Install grype curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin # Scan and fail on high/critical /usr/local/bin/grype \ ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \ --fail-on high \ -o json > scan-results.json # 5. Generate SBOM - name: Generate SBOM run: | # Install syft curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin # Generate SBOM in SPDX format /usr/local/bin/syft \ ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \ -o spdx-json > sbom.spdx.json # Also generate in CycloneDX format /usr/local/bin/syft \ ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \ -o cyclonedx-json > sbom.cyclonedx.json # 6. Login to registry - name: Log in to Container Registry uses: docker/login-action@v2 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} # 7. Push image - name: Push image to registry uses: docker/build-push-action@v5 with: context: . push: true tags: | ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:v${{ github.ref_name }} # 8. Install cosign - name: Install cosign uses: sigstore/cosign-installer@v3 with: cosign-release: 'v2.2.0' # 9. Sign image with cosign (OIDC keyless on GitHub) - name: Sign image with cosign run: | # GitHub Actions automatically provides OIDC token cosign sign --yes \ ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.push.outputs.digest }} # 10. Attach SBOM attestation - name: Attach SBOM to image run: | cosign attach attestation --yes \ --attestation sbom.spdx.json \ ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.push.outputs.digest }} # 11. Create SLSA provenance - name: Generate SLSA provenance uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3@v1.9.0 with: image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} digest: ${{ steps.push.outputs.digest }} registry-username: ${{ github.actor }} registry-password: ${{ secrets.GITHUB_TOKEN }} # 12. Upload scan results - name: Upload vulnerability scan results uses: actions/upload-artifact@v4 if: always() with: name: vulnerability-scan path: scan-results.json # 13. Upload SBOM - name: Upload SBOM uses: actions/upload-artifact@v4 with: name: sbom path: | sbom.spdx.json sbom.cyclonedx.json # 14. Create GitHub release (on main branch only) - name: Create Release if: github.ref == 'refs/heads/main' && github.event_name == 'push' uses: softprops/action-gh-release@v1 with: files: | sbom.spdx.json sbom.cyclonedx.json scan-results.json tag_name: v${{ github.run_number }} body: | Image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} SBOM: sbom.spdx.json Scan Results: scan-results.json draft: false prerelease: false env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # 15. Notify deployment system - name: Notify ArgoCD if: github.ref == 'refs/heads/main' && github.event_name == 'push' run: | curl -X POST https://argocd.company.com/api/webhooks/github \ -H "Authorization: Bearer ${{ secrets.ARGOCD_TOKEN }}" \ -H "Content-Type: application/json" \ -d '{ "image": "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}", "digest": "${{ steps.push.outputs.digest }}", "sbom": "sbom.spdx.json" }'Key Features of This Workflow
Feature | Benefit |
|---|---|
OIDC keyless signing | No secrets stored; uses GitHub's OIDC identity |
Vulnerability scanning | Grype scans and fails the build on high/critical |
SBOM generation | Two formats (SPDX, CycloneDX) for compliance |
SLSA provenance | Cryptographic proof of build chain |
Attestation | SBOM attached to image in registry |
Multi-arch tags | Supports latest, version, SHA tags |
Artifact upload | Scan results and SBOM available for audit |
ArgoCD notification | Automatic deployment trigger on main branch |
Part 4: GitLab CI — Complete Pipeline
# .gitlab-ci.yml variables: REGISTRY: registry.gitlab.com IMAGE_NAME: $CI_PROJECT_PATH DOCKER_DRIVER: overlay2 stages: - verify - build - scan - sign - deploy # Stage 1: Verify base imageverify_base_image: stage: verify image: alpine:latest script: - apk add --no-cache docker curl - docker pull cleanstart/postgresql:15-prod@sha256:abc123... - | docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ anchore/grype:latest \ cleanstart/postgresql:15-prod@sha256:abc123... \ --fail-on medium only: - main - develop # Stage 2: Build imagebuild_image: stage: build image: docker:latest services: - docker:dind script: - docker build -t $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA . - docker tag $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA $REGISTRY/$IMAGE_NAME:latest artifacts: reports: container_scanning: scan-results.json only: - main - develop # Stage 3: Scan for vulnerabilitiesscan_image: stage: scan image: anchore/grype:latest script: - | grype \ $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA \ --fail-on high \ -o json > scan-results.json - grype $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA --fail-on high artifacts: paths: - scan-results.json expire_in: 30 days allow_failure: false only: - main - develop # Stage 4: Generate SBOMgenerate_sbom: stage: scan image: anchore/syft:latest script: - syft $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA -o spdx-json > sbom.spdx.json - syft $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA -o cyclonedx-json > sbom.cyclonedx.json artifacts: paths: - sbom.*.json expire_in: 90 days only: - main - develop # Stage 5: Sign image with cosignsign_image: stage: sign image: gcr.io/projectsigstore/cosign:latest script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - | cosign sign --key $COSIGN_KEY \ $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA - | cosign attach attestation --attestation sbom.spdx.json \ $REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA artifacts: paths: - sbom.*.json expire_in: 90 days only: - main environment: name: production kubernetes: namespace: default # Stage 6: Deploy to productiondeploy_production: stage: deploy image: bitnami/kubectl:latest script: - kubectl set image deployment/app app=$REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA --record - kubectl rollout status deployment/app environment: name: production kubernetes: namespace: production only: - main when: manualPart 5: Jenkins — Complete Jenkinsfile
// Jenkinsfile @Library('shared-library') _ pipeline { agent any options { buildDiscarder(logRotator(numToKeepStr: '10')) timeout(time: 30, unit: 'MINUTES') timestamps() } environment { REGISTRY = credentials('docker-registry-url') IMAGE_NAME = "${env.BUILD_TAG.toLowerCase()}" COSIGN_KEY = credentials('cosign-private-key') COSIGN_PASSWORD = credentials('cosign-password') } stages { stage('Checkout') { steps { checkout scm script { env.GIT_COMMIT_SHORT = sh( script: "git rev-parse --short HEAD", returnStdout: true ).trim() } } } stage('Verify Base Image') { steps { script { sh ''' docker pull cleanstart/postgresql:15-prod@sha256:abc123... docker run --rm anchore/grype:latest \ cleanstart/postgresql:15-prod@sha256:abc123... \ --fail-on medium ''' } } } stage('Build Image') { steps { script { sh ''' docker build \ -t ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} \ -t ${REGISTRY}/${IMAGE_NAME}:latest \ . ''' } } } stage('Scan for Vulnerabilities') { steps { script { sh ''' docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ anchore/grype:latest \ ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} \ --fail-on high \ -o json > scan-results.json ''' } archiveArtifacts artifacts: 'scan-results.json', allowEmptyArchive: false } } stage('Generate SBOM') { steps { script { sh ''' docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ anchore/syft:latest \ ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} \ -o spdx-json > sbom.spdx.json docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ anchore/syft:latest \ ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} \ -o cyclonedx-json > sbom.cyclonedx.json ''' } archiveArtifacts artifacts: 'sbom.*.json', allowEmptyArchive: false } } stage('Push Image') { steps { script { sh ''' docker push ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} docker push ${REGISTRY}/${IMAGE_NAME}:latest ''' } } } stage('Sign Image') { steps { script { sh ''' export COSIGN_EXPERIMENTAL=1 cosign sign --key ${COSIGN_KEY} \ ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} ''' } } } stage('Attach Attestation') { steps { script { sh ''' cosign attach attestation --attestation sbom.spdx.json \ ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} ''' } } } stage('Deploy to Production') { when { branch 'main' } steps { script { sh ''' kubectl set image deployment/app \ app=${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} \ --record kubectl rollout status deployment/app ''' } } } } post { always { // Clean up sh 'docker rmi ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER} || true' } success { // Notify sh ''' curl -X POST https://slack.company.com/api/chat.postMessage \ -H "Authorization: Bearer ${SLACK_TOKEN}" \ -d "channel=#devops&text=Image pushed: ${REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER}" ''' } failure { // Alert sh ''' curl -X POST https://slack.company.com/api/chat.postMessage \ -H "Authorization: Bearer ${SLACK_TOKEN}" \ -d "channel=#devops&text=Build failed: ${BUILD_URL}" ''' } }}Part 6: Google Cloud Build — Complete cloudbuild.yaml
# cloudbuild.yaml steps: # Step 1: Verify base image - name: gcr.io/cloud-builders/docker id: verify-base entrypoint: bash args: - -c - | docker pull cleanstart/postgresql:15-prod@sha256:abc123... # Verify signature (requires cosign) docker run --rm gcr.io/projectsigstore/cosign:latest \ verify --key https://keys.cleanstart.dev/cosign.pub \ cleanstart/postgresql:15-prod@sha256:abc123... # Step 2: Build image - name: gcr.io/cloud-builders/docker id: build args: - build - -t - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} - -t - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:latest - . # Step 3: Scan with Artifact Registry - name: gcr.io/cloud-builders/container-scanning id: scan args: - scan - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} # Step 4: Push to Artifact Registry - name: gcr.io/cloud-builders/docker id: push args: - push - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:latest # Step 5: Generate SBOM with syft - name: gcr.io/cloud-builders/container-analysis id: sbom args: - sbom - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} # Step 6: Sign image with KMS key - name: gcr.io/cloud-builders/gke-deploy id: sign entrypoint: bash args: - -c - | docker run --rm \ -v /var/run/docker.sock:/var/run/docker.sock \ gcr.io/projectsigstore/cosign:latest sign \ --key gcpkms://projects/${PROJECT_ID}/locations/${_KMS_REGION}/keyRings/${_KMS_KEYRING}/cryptoKeys/${_KMS_KEY} \ ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} # Step 7: Deploy to GKE - name: gcr.io/cloud-builders/gke-deploy id: deploy args: - run - --filename=k8s/ - --image=${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} - --location=${_GKE_REGION} - --cluster=${_GKE_CLUSTER} - --namespace=production images: - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${SHORT_SHA} - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:latest substitutions: _REGION: us-central1 _REPOSITORY: docker-repo _IMAGE_NAME: app _KMS_REGION: us-central1 _KMS_KEYRING: cloudbuild-keys _KMS_KEY: image-signing-key _GKE_REGION: us-central1 _GKE_CLUSTER: production-cluster options: logging: CLOUD_LOGGING_ONLY machineType: N1_HIGHCPU_8 tags: - build - sign - deploy - gkePart 7: Multi-Environment Pipeline
Deploy the same image to different environments with different ConfigMaps:
# .github/workflows/deploy.yml (separate from build.yml) name: Deploy to Environment on: workflow_dispatch: inputs: environment: description: 'Target environment' required: true type: choice options: - staging - production image_tag: description: 'Image tag to deploy' required: true default: 'latest' jobs: deploy: runs-on: ubuntu-latest environment: ${{ github.event.inputs.environment }} permissions: contents: read packages: read steps: - name: Checkout uses: actions/checkout@v4 - name: Configure kubectl run: | echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > kubeconfig.yaml export KUBECONFIG=$(pwd)/kubeconfig.yaml - name: Deploy to ${{ github.event.inputs.environment }} run: | # Verify image signature cosign verify --key ${{ secrets.COSIGN_PUBLIC_KEY }} \ ghcr.io/${{ github.repository }}:${{ github.event.inputs.image_tag }} # Deploy with environment-specific ConfigMap kubectl apply -f k8s/${{ github.event.inputs.environment }}/configmap.yaml kubectl set image deployment/app \ app=ghcr.io/${{ github.repository }}:${{ github.event.inputs.image_tag }} \ -n ${{ github.event.inputs.environment }} # Wait for rollout kubectl rollout status deployment/app -n ${{ github.event.inputs.environment }} --timeout=5m - name: Verify deployment run: | # Health checks kubectl get pods -n ${{ github.event.inputs.environment }} kubectl describe deployment app -n ${{ github.event.inputs.environment }}Part 8: Automated Base Image Updates
When CleanStart releases a new version, automatically rebuild:
# .github/workflows/renovate.json (Renovate Bot config) { "extends": ["config:base"], "docker": { "enabled": true, "automerge": false, "major": { "enabled": false } }, "customManagers": [ { "customType": "dockerfile", "fileMatch": ["^Dockerfile$"], "datasourceTemplate": "docker", "matchStrings": [ "FROM cleanstart/(?<depName>[a-z-]+):(?<currentValue>[0-9.-]+)(?<distroImage>-prod)?" ], "versioningTemplate": "semver" } ], "automergeType": "pr", "automergeStrategy": "squash", "semanticCommits": "enabled", "schedule": ["weekly"], "platformAutomerge": true, "prCreation": "not-pending"}When Renovate detects a new CleanStart base image, it creates a PR updating the Dockerfile. It runs the build pipeline (scan, sign, test). If all checks pass, it auto-merges the PR. GitHub Actions deploys the updated image.
Part 9: Pipeline Security Best Practices
1. Pin Image Digests
# WRONG: Uses mutable tagFROM cleanstart/postgresql:15-prod # CORRECT: Uses immutable digestFROM cleanstart/postgresql:15-prod@sha256:abc123def456...Digests cannot change; tags can be retagged to different images.
2. Verify Signatures Before Building
# In pipeline, before using a base imagecosign verify --key https://keys.cleanstart.dev/cosign.pub \ cleanstart/postgresql:15-prod@sha256:abc123... # If verification fails, the pipeline should stop3. Never Skip Scanning
# WRONGgrype image:latest || true # Ignores failures # CORRECTgrype image:latest --fail-on high# Exit code non-zero = pipeline fails4. Store Signing Keys Securely
Never commit private keys to Git:
# WRONGgit add cosign.keygit commit -m "Add signing key" # CORRECT# Store in GitHub Secrets, Cloud Build KMS, or similar# Reference via ${{ secrets.COSIGN_KEY }}In GitHub Actions:
- name: Sign image run: cosign sign --key ${{ secrets.COSIGN_KEY }} image:tagIn Google Cloud Build:
steps: - name: gcr.io/projectsigstore/cosign entrypoint: | cosign sign --key gcpkms://projects/PROJECT_ID/locations/us-central1/keyRings/ring/cryptoKeys/key image5. Enable Audit Logging
Every pipeline run should be auditable. GitHub Actions automatically logs builds. Jenkins requires enabling and configuring audit plugins. GitLab enables audit event logging. Cloud Build enables Cloud Logging. Query audit logs when needed:
gcloud logging read "resource.type=cloud-build"6. Require Manual Approval for Production
# GitHub Actionsdeploy_production: environment: name: production # Requires manual approval before running// Jenkinsinput "Deploy to production?"Summary: The Production CI/CD Pipeline
Stage | Tool | Gate | Output |
|---|---|---|---|
Verify base | cosign | Fail if signature invalid | Baseline verified |
Build | docker build | Fail if build fails | Image created |
Scan | grype | Fail if high/critical | Vulnerability report |
Generate | syft | Warn if incomplete | SBOM (SPDX, CycloneDX) |
Push | docker push | Fail if push fails | Image in registry |
Sign | cosign | Fail if signing fails | Signed image |
Attest | cosign | Warn if attestation fails | SBOM attached |
Deploy | kubectl | Fail if rollout times out | Live in production |
This pipeline ensures every image is built from version-controlled source, every image is scanned and signed, every image has a complete SBOM and provenance, every image is traceable to a Git commit, and every deployment is auditable and reproducible.
This meets SLSA Level 3 requirements and aligns with NIST 800-190, CIS Docker Benchmark, and FedRAMP supply chain security standards.