CleanStart uses an image-based upgrade model. When a CVE is fixed, CleanStart publishes a new image version. You pull the new image, rebuild your application image, and redeploy. There is no in-place patching of running containers. This section covers the complete upgrade workflow, from monitoring for new releases through production verification.
The following diagram illustrates the patching workflow from CVE detection through production deployment:
graph TD A["CVE<br/>Published"] -->|Alert| B["Monitor<br/>Registry<br/>Renovate/Dependabot"] B -->|Detect| C["CleanStart<br/>New Release<br/>3.12.5-prod"] C -->|Trigger| D["Rebuild<br/>Application<br/>Image"] D -->|Fetch| E["Pull<br/>cleanstart-python:3.12.5-prod"] E -->|Run| F["docker build<br/>-t myapp:v2.1"] F -->|Generate| G["Build<br/>Artifacts<br/>SBOM + Provenance"] G -->|Test| H["Run<br/>Acceptance<br/>Tests"] H -->|Test Results| I{All Tests<br/>Pass?} I -->|No| J["Fix Issues<br/>Rebuild"] J -->|Iterate| F I -->|Yes| K["Tag<br/>Image"] K -->|Deploy| L["Dev<br/>Environment"] L -->|Verify| M["Smoke<br/>Tests"] M -->|Monitor| N{Stable<br/>?} N -->|No| O["Rollback<br/>Previous"] N -->|Yes| P["Deploy<br/>Staging"] P -->|Full Test| Q["Integration<br/>Tests"] Q -->|Manual| R{Approve<br/>Prod?} R -->|No| O R -->|Yes| S["Deploy<br/>Production"] S -->|Monitor| T["Health<br/>Checks"] T -->|Success| U["Patching<br/>Complete"] style A fill:#ff9999 style C fill:#ffff99 style D fill:#ccffcc style G fill:#ccffcc style U fill:#99ff99CleanStart Upgrade Model
How It Works
CleanStart publishes fixed images by rebuilding the affected base image from verified source with the security patch applied. Each release gets a version tag following semantic versioning with an environment suffix.
Image naming convention:
registry.cleanstart.com/<runtime>:<major>.<minor>.<patch>-<environment>Examples:
registry.cleanstart.com/python3:3.12.5-prodregistry.cleanstart.com/python3:3.12.4-prodregistry.cleanstart.com/nodejs:20.10.0-prodregistry.cleanstart.com/go:1.21.5-prodregistry.cleanstart.com/rust:1.73.0-prodregistry.cleanstart.com/java11:11.0.21-prodWhen a CVE affects Python 3.12, CleanStart:
- Identifies affected base image versions
- Rebuilds image with patched packages
- Tests and verifies the build (cosign signature included)
- Publishes as new semantic version (e.g.,
3.12.4-prod→3.12.5-prod) - Broadcasts release via registry notifications (webhooks, RSS feed)
Image Repository Locations
Registry | Purpose | Usage |
|---|---|---|
| Container images | Pull in Dockerfile: |
| APK packages | Install additional packages at build time |
Ghcr.io (mirror) | GitHub Container Registry | Fallback/mirror access |
Supporting Packages
Additional APK packages come from clnpkgs.clnstrt.dev. When building your app image, you can add packages:
FROM registry.cleanstart.com/python3:3.12.5-prod # Install additional packagesRUN --mount=type=cache,target=/var/cache/apk \ apk add --no-cache \ curl@=8.5.0-r0 \ openssl@=3.1.4-r5 \ postgresql-client@=15.2-r0All packages are scanned and verified. Version pinning (shown above with @=version-r<revision>) is recommended.
Version Pinning Strategy
Digest vs Tag
Semantic version tags (mutable, recommended for most teams):
FROM registry.cleanstart.com/python3:3.12.5-prodPro: Simple to read and maintain; Renovate/Dependabot automatically identify upgrades. Con: Technically mutable (though CleanStart images are immutable in practice).
Image digests (immutable, for regulatory/compliance teams):
FROM registry.cleanstart.com/python3@sha256:abc123def456...Pro: Guarantees exact image used; tamper-proof audit trail. Con: Hard to read; requires tooling to discover and update digest for upgrades.
Hybrid approach (recommended for production):
FROM registry.cleanstart.com/python3:3.12.5-prod@sha256:abc123def456...Combines readability (tag) with immutability guarantee (digest). Renovate supports this format.
Monitoring for New Releases
Option 1: Renovate Bot (Recommended)
Renovate automatically detects new CleanStart image versions and opens pull requests.
Setup in .renovaterc.json:
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ "config:base", ":dependencyDashboard", ":semanticCommits" ], "registryAliases": { "cleanstart": "registry.cleanstart.com" }, "dockerfile": { "enabled": true, "automerge": false, "schedule": ["before 3am on Monday"] }, "packageRules": [ { "description": "CleanStart base image updates", "matchDatasources": ["docker"], "matchPackageNames": [ "registry.cleanstart.com/python3", "registry.cleanstart.com/nodejs", "registry.cleanstart.com/go" ], "schedule": ["before 3am on Monday"], "automerge": false, "labels": ["dependencies", "security"] }, { "description": "Patch-level updates (auto-merge)", "matchDatasources": ["docker"], "matchUpdateTypes": ["patch"], "automerge": true, "automergeSchedule": ["after 10pm before 6am on weekdays"] } ]}What Renovate does: Monitors registry.cleanstart.com registry for new tags. Opens PR with updated FROM line and Dockerfile digest. Runs CI tests automatically. Optionally auto-merges patch updates (3.12.4 → 3.12.5). Sends Slack notifications for new major/minor versions.
Option 2: Dependabot
GitHub's built-in tool (simpler than Renovate but fewer options).
Setup in .github/dependabot.yml:
version: 2updates: - package-ecosystem: "docker" directory: "/" schedule: interval: "weekly" day: "monday" time: "02:00" open-pull-requests-limit: 5 reviewers: - "security-team" labels: - "dependencies" - "security" commit-message: prefix: "chore(deps):" pull-request-branch-name: separator: "/" include: "dependency"Option 3: Manual Registry Polling
Set up a simple cron job to check for new releases:
#!/bin/bash# check_cleanstart_updates.sh # Fetch latest tags from CleanStart registrycheck_image() { local image=$1 echo "Checking $image..." # Docker CLI method docker pull --dry-run "$image" 2>&1 | grep -q "Downloaded" && { echo "New version available: $image" # Send Slack/email notification curl -X POST $SLACK_WEBHOOK \ -H 'Content-type: application/json' \ --data "{\"text\": \"CleanStart update available: $image\"}" }} check_image "registry.cleanstart.com/python3:latest"check_image "registry.cleanstart.com/nodejs:latest"check_image "registry.cleanstart.com/go:latest"Run daily via cron:
0 2 * * * /usr/local/bin/check_cleanstart_updates.shCVE Response Timeline
When a CVE affects CleanStart base images:
T+0 (CVE Disclosure)
CVE published (NVD, vendor advisories). CleanStart security team notified. Assessment of affected images begins.
T+2-4 hours (Internal Build)
Patch identified and tested. Affected base image rebuilt with patched packages. Internal testing: functionality + performance. Image signed with cosign key.
T+4-8 hours (Publication)
New image version published to registry.cleanstart.com. Release notes posted with CVE details and fixed version. Webhook events sent to registries monitoring CleanStart. Slack channel notification for customers with active subscriptions.
Customer Action (Same Day to Next Business Day)
Day 0 (within 4-8 hours):
# Renovate/Dependabot automatically opens PR# OR manual workflow: # 1. Update Dockerfile with new image versionsed -i 's/3.12.4-prod/3.12.5-prod/g' Dockerfile # 2. Rebuild and test locallydocker build -t myapp:new-patch -f Dockerfile .docker run --rm myapp:new-patch pytest # 3. Push to repo (triggers CI/CD)git commit -am "chore(deps): update CleanStart base image to fix CVE"git push origin patch-cve-fixDay 0-1 (CI/CD): CI pipeline detects commit. Builds new application image. Runs vulnerability scan (Trivy or Grype). Performs regression tests. If all pass, image tagged and pushed to application registry.
Day 1-2 (Deployment): Image pulled and deployed to staging. Canary/progressive rollout to production. Monitoring confirms no regressions. Complete.
Image Version Bumps in Dockerfile
Before: Python 3.12.4
FROM registry.cleanstart.com/python3:3.12.4-prod@sha256:abc123def456... WORKDIR /appCOPY requirements.txt .RUN --mount=type=cache,target=/var/cache/apk \ apk add --no-cache python3-pip && \ pip install --no-cache-dir -r requirements.txt COPY . .CMD ["python3", "main.py"]After: Python 3.12.5 (CVE Fix)
FROM registry.cleanstart.com/python3:3.12.5-prod@sha256:xyz789abc111... WORKDIR /appCOPY requirements.txt .RUN --mount=type=cache,target=/var/cache/apk \ apk add --no-cache python3-pip && \ pip install --no-cache-dir -r requirements.txt COPY . .CMD ["python3", "main.py"]Changes: Tag: 3.12.4-prod → 3.12.5-prod. Digest: sha256:abc123... → sha256:xyz789... (because image changed).
Kubernetes Rolling Update Strategy
When you have CleanStart-based images deployed in Kubernetes, rolling updates minimize downtime.
Helm Values Update
Before (values.yaml):
image: registry: registry.cleanstart.com repository: python3 tag: "3.12.4-prod" pullPolicy: IfNotPresent replicaCount: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0After (values.yaml):
image: registry: registry.cleanstart.com repository: python3 tag: "3.12.5-prod" pullPolicy: IfNotPresent replicaCount: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0Deploy with Helm
# Dry-run to verify changeshelm upgrade --install myapp ./chart \ --values values.yaml \ --dry-run \ --debug # Actually upgradehelm upgrade --install myapp ./chart \ --values values.yaml \ --wait \ --timeout 10m # Check rollout statuskubectl rollout status deployment/myappKubernetes Deployment YAML
apiVersion: apps/v1kind: Deploymentmetadata: name: myapp namespace: defaultspec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 selector: matchLabels: app: myapp template: metadata: labels: app: myapp annotations: # Force pod recreation on image change deployment.kubernetes.io/revision: "2" spec: imagePullPolicy: IfNotPresent containers: - name: myapp image: registry.cleanstart.com/python3:3.12.5-prod resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 5 periodSeconds: 5Rollout sequence:
- Pod 1 (replica-0) shuts down gracefully (30s termination grace period)
- New Pod 1 starts with new image (3.12.5)
- Pod 1 becomes Ready (passes readiness probe)
- Pod 2 (replica-1) cycles similarly
- Pod 3 (replica-2) cycles similarly
- All replicas now running 3.12.5
Verify status:
kubectl rollout status deployment/myapp -wkubectl get pods -l app=myapp -o widekubectl describe deployment myappCI/CD Pipeline Integration
GitHub Actions Example
.github/workflows/image-update-cve-patch.yml:
name: CVE Patch - Build and Deploy on: pull_request: paths: - "Dockerfile" - "src/**" - ".github/workflows/image-update-cve-patch.yml" workflow_dispatch: env: REGISTRY: ghcr.io IMAGE_NAME: ${{ github.repository }}/myapp jobs: build-and-test: runs-on: ubuntu-latest permissions: contents: read packages: write steps: - uses: actions/checkout@v4 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Log in to Container Registry uses: docker/login-action@v3 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Extract metadata id: meta uses: docker/metadata-action@v5 with: images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} tags: | type=ref,event=branch type=sha,prefix={{branch}}- type=semver,pattern={{version}} type=raw,value=latest,enable={{is_default_branch}} - name: Build and push Docker image uses: docker/build-push-action@v5 with: context: . push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max scan-vulnerability: runs-on: ubuntu-latest needs: build-and-test steps: - name: Run Trivy vulnerability scanner uses: aquasecurity/trivy-action@master with: image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} format: "sarif" output: "trivy-results.sarif" - name: Upload Trivy results to GitHub Security uses: github/codeql-action/upload-sarif@v2 if: always() with: sarif_file: "trivy-results.sarif" deploy-staging: runs-on: ubuntu-latest needs: scan-vulnerability if: github.event_name == 'pull_request' steps: - uses: actions/checkout@v4 - name: Update Helm values for staging run: | sed -i "s/tag: .*/tag: \"${{ github.sha }}\"/g" helm/values-staging.yaml - name: Deploy to staging with Helm uses: actions/helm@v1 with: release: "myapp-staging" namespace: "staging" chart: "./helm" values: "./helm/values-staging.yaml" deploy-production: runs-on: ubuntu-latest needs: scan-vulnerability if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: production steps: - uses: actions/checkout@v4 - name: Deploy to production with Helm uses: actions/helm@v1 with: release: "myapp" namespace: "production" chart: "./helm" values: "./helm/values-prod.yaml" - name: Verify rollout run: | kubectl rollout status deployment/myapp -n production --timeout=10m - name: Slack notification if: always() uses: slackapi/slack-github-action@v1.24.0 with: webhook-url: ${{ secrets.SLACK_WEBHOOK }} payload: | { "text": "Production deployment: ${{ job.status }}", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "*Deployment Status: ${{ job.status }}*\n${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" } } ] }GitLab CI Example
.gitlab-ci.yml:
stages: - build - scan - deploy-staging - deploy-prod variables: REGISTRY: registry.gitlab.com IMAGE_NAME: $CI_PROJECT_PATH/myapp IMAGE_TAG: $CI_COMMIT_SHORT_SHA build-image: stage: build image: docker:latest services: - docker:dind script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $REGISTRY - docker build -t $REGISTRY/$IMAGE_NAME:$IMAGE_TAG -t $REGISTRY/$IMAGE_NAME:latest . - docker push $REGISTRY/$IMAGE_NAME:$IMAGE_TAG - docker push $REGISTRY/$IMAGE_NAME:latest only: - branches scan-trivy: stage: scan image: aquasec/trivy:latest script: - trivy image --exit-code 0 --severity HIGH,CRITICAL $REGISTRY/$IMAGE_NAME:$IMAGE_TAG artifacts: reports: container_scanning: trivy-report.json deploy-staging: stage: deploy-staging image: alpine/helm:latest script: - helm repo add cleanstart registry.cleanstart.com - helm repo update - helm upgrade --install myapp ./helm --namespace staging --values helm/values-staging.yaml --set image.tag=$IMAGE_TAG --wait environment: name: staging url: https://staging.myapp.example.com only: - branches deploy-production: stage: deploy-prod image: alpine/helm:latest script: - helm upgrade --install myapp ./helm --namespace production --values helm/values-prod.yaml --set image.tag=$IMAGE_TAG --wait - kubectl rollout status deployment/myapp -n production environment: name: production url: https://myapp.example.com only: - main when: manualVerifying New Images
Cosign Verification
CleanStart images are signed with cosign. Verify before deployment:
# Install cosign (if needed)curl -Lo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64chmod +x /usr/local/bin/cosign # Verify signaturecosign verify \ --certificate-identity-regexp "https://github.com/cleanstart/.*" \ --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ registry.cleanstart.com/python3:3.12.5-prod # Expected output:# Verification successful!# {cert-info...}In your Dockerfile (optional but recommended):
# Multi-stage: verify then useFROM registry.cleanstart.com/python3:3.12.5-prod as verifyRUN cosign verify \ --certificate-identity-regexp "https://github.com/cleanstart/.*" \ --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ registry.cleanstart.com/python3:3.12.5-prod FROM registry.cleanstart.com/python3:3.12.5-prod# ... rest of DockerfileSBOM Comparison (Old vs New Image)
Extract and compare SBOMs to understand what changed:
# Generate SBOM for old imagesyft registry.cleanstart.com/python3:3.12.4-prod -o json > old.sbom.json # Generate SBOM for new imagesyft registry.cleanstart.com/python3:3.12.5-prod -o json > new.sbom.json # Diff (new packages, removed packages, version changes)diff <(jq -r '.artifacts[].name' old.sbom.json | sort) \ <(jq -r '.artifacts[].name' new.sbom.json | sort)Example output (additions/removals):
< curl< openssl---> curl> openssl-3.1.5-r0> alpine-baselayout-3.6.0-r0Emergency Hotfix Procedure
Critical CVE affecting production? Reduce the timeline from days to hours.
T+0 (Alert)
- Security team confirms severity (CVSS > 8.0)
- Checks if CleanStart images affected
- Opens urgent Slack channel:
#security-incident-cve-xxxx
T+15 minutes (Triage)
# Quick check: is your production image affected?docker pull registry.cleanstart.com/python3:3.12.4-prodgrype registry.cleanstart.com/python3:3.12.4-prod | grep CVE-2024-XXXXX # Result: YES or NOT+30 minutes (Prepare)
CleanStart publishes fixed image: 3.12.5-hotfix-prod. You update Dockerfile + commit to hotfix branch:.
git checkout -b hotfix/cve-2024-xxxxxsed -i 's/3.12.4-prod/3.12.5-hotfix-prod/g' Dockerfilegit commit -am "hotfix: apply critical CVE patch"git push origin hotfix/cve-2024-xxxxxT+45 minutes (Build + Test)
CI/CD auto-triggers on hotfix branch. Build, scan, and deploy to staging immediately (skip some slow tests). Quick smoke tests in staging.
T+60 minutes (Production Deploy)
# Helm upgrade with bypass of change control (emergency approval)helm upgrade myapp ./helm \ --values values.yaml \ --set image.tag=3.12.5-hotfix-prod \ --wait \ --timeout 5m \ --atomic # Monitor immediatelykubectl logs -f deployment/myapp -n productionkubectl top pods -l app=myappT+90 minutes (Verification + Comms)
All 3 replicas running, healthy. No error spikes in logs. Send Slack: "CVE-2024-XXXXX patched and deployed to production".
Parallel vs Sequential: Never run multiple hotfixes in parallel. Always wait for verification before proceeding. Use --atomic flag to auto-rollback if probe fails.
Multi-Image Coordination
Your stack may use multiple CleanStart base images. A typical application stack might include: python3:3.12.4-prod for the API service, nodejs:20.10.0-prod for the frontend, go:1.21.5-prod for background workers, and java11:11.0.21-prod for batch processors.
CVE affects OpenSSL → all 4 images need patches. Coordinate:
Approach 1: Parallel (Fastest)
Update all 4 Dockerfiles at once. Merge single PR. Deploy all 4 apps simultaneously.
# Dockerfile.api (Python)FROM registry.cleanstart.com/python3:3.12.5-prod # UPDATED # Dockerfile.frontend (Node)FROM registry.cleanstart.com/nodejs:20.10.1-prod # UPDATED # Dockerfile.worker (Go)FROM registry.cleanstart.com/go:1.21.6-prod # UPDATED # Dockerfile.batch (Java)FROM registry.cleanstart.com/java11:11.0.22-prod # UPDATEDgit checkout -b security/openssl-cve-patch-all# Update all 4 Dockerfilesgit commit -am "security: apply OpenSSL CVE patch across all services"git push origin security/openssl-cve-patch-all# Single PR, single CI/CD run, deploy all services at onceApproach 2: Sequential (Safer)
Update one image at a time. Stagger deployments by 1-2 hours. Easier to isolate issues.
# Hour 0: API service (Python)git checkout -b security/openssl-cve-apised -i 's/3.12.4-prod/3.12.5-prod/g' Dockerfile.apigit commit && git push # Hour 1-2: Frontend service (Node)git checkout -b security/openssl-cve-frontendsed -i 's/20.10.0-prod/20.10.1-prod/g' Dockerfile.frontendgit commit && git push # Hour 3-4: Worker service (Go)# ... and so onRecommendation: Use parallel approach for patch-level updates. Use sequential for major version upgrades where behavior might change.
Monitoring and Alerting for Updates
Prometheus Alert Rules
alerts.yml:
groups: - name: cleanstart interval: 30s rules: - alert: OutdatedCleanStartImage expr: | time() - cleanstart_image_build_timestamp > 2592000 for: 1h labels: severity: warning annotations: summary: "Running outdated CleanStart image" description: "Container {{ $labels.pod }} has image built > 30 days ago" - alert: VulnerablePackageDetected expr: | vulnerabilities_found{source="grype"} > 0 for: 15m labels: severity: critical annotations: summary: "Vulnerability detected in running image" description: "{{ $value }} vulnerabilities found in {{ $labels.image }}" - alert: ImagePullFailed expr: | increase(pod_image_pull_errors_total[5m]) > 0 labels: severity: critical annotations: summary: "Failed to pull CleanStart image" description: "Cannot pull {{ $labels.image }}"Deploy Alert Rules to Kubernetes
kubectl apply -f alerts.yml -n monitoringRollback Procedure
If new image causes issues:
# Identify previous working revisionhelm history myapp -n production # Output:# REVISION UPDATED STATUS CHART APP VERSION# 1 Wed Mar 19 10:30:00 2025 superseded myapp-1.0.0 1.0.0# 2 Wed Mar 20 14:45:00 2025 deployed myapp-1.0.1 1.0.1 <-- CURRENT (BROKEN) # Rollback to revision 1helm rollback myapp 1 -n production --wait # Verifykubectl rollout status deployment/myapp -n productionSee the Rollback and Disaster Recovery guide for detailed procedures.
Summary
Image-based upgrades provide: Simplicity: Pull new image, rebuild app image, redeploy. Speed: No patching delays; fixes available same day as CVE. Auditability: Every version change is tracked in git history. Reproducibility: Dockerfile + image tag exactly defines runtime.
Use Renovate or Dependabot to automate detection. Test in CI/CD. Deploy with Helm rolling updates. Verify with cosign and SBOM comparison. In emergencies, use hotfix procedures to deploy in under 90 minutes.
Next Steps: Secure Operations at Scale
Understand the supply chain — Know what you're upgrading: What is Supply Chain Security? — Why every upgrade matters. What is a CVE? — Understand vulnerabilities you're patching. Machine Speed vs Human Speed — Why fast patching is critical.
Implement secure builds — Ensure images are safe before deployment: Build Stage Security — Secure build process for app images. The Continuous Trust Loop — Automated rebuilds for CVEs. Verified Source Philosophy — Architecture of secure base images.
Operational best practices — Manage images in production: Helm Charts & Kubernetes — Deploy securely with Helm. Network Policies — Control traffic between services. Secret Management — Secure credential handling. Multi-Cloud Registry Operations — Manage images across environments. Supply Chain Disaster Recovery — Protect your supply chain.