Purpose

When you upgrade CleanStart image versions in production, you want controlled, observable rollouts—not big-bang deployments that risk production stability. This guide covers canary deployments, blue-green deployments, and progressive delivery strategies specifically designed for CleanStart image upgrades.

A new image version is not just a version bump: it contains new OS packages (from apk package manager), new system libraries, potentially new runtimes, and new security patches. Even with CleanStart's 78-test inspection suite and your own acceptance tests, production traffic behaves differently than test environments.

Progressive delivery gives you real production signal before committing to a full rollout. If you see errors, latency spikes, or unexpected resource usage, you can roll back quickly with minimal blast radius.

Why Progressive Delivery Matters for Image Upgrades

Risk Categories When Upgrading Images

When you upgrade from one CleanStart image version to another, you're not just updating a version number—you're changing the entire runtime environment. The new image version includes updated OS packages, new system libraries, potentially new language runtime versions, and new security patches. Even with CleanStart's rigorous testing and your own acceptance tests, production traffic exposes edge cases that test environments don't reveal.

Runtime incompatibility emerges when fundamental components change their behavior in subtle ways. A new Go version may compile binaries with different optimization strategies that expose latent concurrency bugs. A new glibc version may have behavioral differences in corner-case scenarios like unusual locale handling or signal delivery. A new OpenSSL version may enforce stricter TLS validation rules, breaking applications that rely on looser RFC compliance. Applications sometimes depend on undocumented behavior of old library versions—behavior that changes with updates. These incompatibilities are invisible in staging environments but manifest under real production load.

Performance regressions represent another class of risk. Updated packages may increase memory footprint, causing pods to exceed memory limits. Garbage collection behavior may change with new runtime versions, causing periodic latency spikes. Startup time may slow due to new initialization code running on each container startup. Security hardening in the base image may reduce throughput on certain workloads that benefit from relaxed security constraints. These performance changes are quantitative, measurable, but only visible under production-like load.

Security edge cases emerge when security patches have unintended consequences. New security patches may break compatibility with old clients or servers that the application communicates with. FIPS 140-3 mode (if enabled) may have unexpected interactions with your application's cryptographic operations. Read-only root filesystem enforcement may expose missing dependencies that were previously hidden by open filesystem access.

Configuration drift occurs when the new image has different characteristics than the old one. If the organization uses cluster autoscaling, a new image version might have a different footprint, affecting how many pods bin-pack onto a single node. If initContainers pull dependencies, they might reference old registry locations that no longer exist. If network policies restrict egress, a new image version might require access to different external services.

A concrete example illustrates why canary deployments matter. You decide to upgrade from CleanStart Python 3.11 to 3.12 because the new version includes security patches and performance improvements. The new version includes OpenSSL 3.x which disables TLS 1.0 by default due to security best practices. Your application works fine with TLS 1.2+, so you expect no issues. However, your application makes calls to a legacy third-party payment gateway that only supports TLS 1.0 due to constraints on their side. You don't discover this incompatibility in staging because the staging environment either doesn't make calls to the live payment gateway, or uses mock credentials that succeed regardless. You only hit this issue after deploying to production. With a canary deployment, you detect the payment gateway failures within the first 5% of traffic, immediately rollback to the old version, and then prepare a proper fix by explicitly re-enabling TLS 1.0 compatibility before rolling out to 100%. Without canary, every customer experiences payment failures until your team realizes the issue and deploys a fix.

Strategy 1: Canary Deployment

Concept

Route a small percentage of traffic to pods running the new image version while keeping most traffic on the stable version. Monitor for errors and degradation. If metrics remain healthy, gradually increase traffic to new version. If problems occur, immediately rollback.

The following diagram shows the canary deployment strategy timeline and traffic shifting:

1Hour 0: Deploy Canary2Hour 1: Monitor3Hour 2: Increase4Hour 3: Monitor5Hour 4: More Traffic6Hour 5: Monitor7Hour 8: Complete Rollout

The timeline for a typical canary deployment spans approximately eight hours. During the first hour, 5% of traffic goes to the canary while it runs on a single pod, with the remaining 90% on nine stable pods. After monitoring for one hour with stable results, you increase traffic to 25% in hour two. Two hours later, around hour four, traffic reaches 50%. By hour four through hour eight, traffic gradually increases until the canary handles 100% of requests. Once the deployment fully completes, you decommission the old version the next day.

Manual Canary with Kubernetes

This approach uses two separate Deployments with a shared Service.

---# Stable deployment (old version) - 9 replicas = 90% trafficapiVersion: apps/v1kind: Deploymentmetadata:  name: api-stable  labels:    app: api    version: stablespec:  replicas: 9  selector:    matchLabels:      app: api      version: stable  template:    metadata:      labels:        app: api        version: stable    spec:      containers:      - name: api        image: registry.cleanstart.com/cleanstart/python:3.11.5        ports:        - containerPort: 8000        livenessProbe:          httpGet:            path: /health            port: 8000          initialDelaySeconds: 10          periodSeconds: 10        readinessProbe:          httpGet:            path: /ready            port: 8000          initialDelaySeconds: 5          periodSeconds: 5        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: 1000m            memory: 512Mi ---# Canary deployment (new version) - 1 replica = 10% trafficapiVersion: apps/v1kind: Deploymentmetadata:  name: api-canary  labels:    app: api    version: canaryspec:  replicas: 1  selector:    matchLabels:      app: api      version: canary  template:    metadata:      labels:        app: api        version: canary    spec:      containers:      - name: api        image: registry.cleanstart.com/cleanstart/python:3.12.1  # new version        ports:        - containerPort: 8000        livenessProbe:          httpGet:            path: /health            port: 8000          initialDelaySeconds: 10          periodSeconds: 10        readinessProbe:          httpGet:            path: /ready            port: 8000          initialDelaySeconds: 5          periodSeconds: 5        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: 1000m            memory: 512Mi ---# Service selects both stable and canaryapiVersion: v1kind: Servicemetadata:  name: apispec:  selector:    app: api  # matches both stable and canary  ports:  - protocol: TCP    port: 80    targetPort: 8000  type: LoadBalancer

Progression Steps:

During hour 0-1, the stable deployment runs with 9 replicas while the canary runs with 1 replica, distributing traffic proportionally. Scale these deployments using kubectl:

kubectl scale deployment api-stable --replicas=9kubectl scale deployment api-canary --replicas=1

In hour 1-2, transition to a 6-4 split:

kubectl scale deployment api-stable --replicas=6kubectl scale deployment api-canary --replicas=4

During hour 2-4, balance traffic at 5-5:

kubectl scale deployment api-stable --replicas=5kubectl scale deployment api-canary --replicas=5

In hour 4-8, shift to 0-10 split by scaling down the stable version:

kubectl scale deployment api-stable --replicas=0kubectl scale deployment api-canary --replicas=10

The next day, once the new version has proven stable, delete the old deployment and rename the canary to stable:

kubectl delete deployment api-stablekubectl patch deployment api-canary --type='json' \  -p='[{"op": "replace", "path": "/metadata/labels/version", "value":"stable"}]'

Automated Canary with Flagger

Flagger automates the traffic shifting based on metrics. This is recommended for most teams.

Installation:

helm repo add flagger https://flagger.apphelm install flagger flagger/flagger \  --namespace flagger-system \  --create-namespace \  --set prometheus.install=true

Canary Configuration:

apiVersion: flagger.app/v1beta1kind: Canarymetadata:  name: apispec:  # Target deployment  targetRef:    apiVersion: apps/v1    kind: Deployment    name: api   # How to route traffic (via service)  service:    port: 80    targetPort: 8000   # Analysis configuration  analysis:    interval: 1m    threshold: 5  # max 5 failed checks before rollback    maxWeight: 100    stepWeight: 10  # increase traffic by 10% every minute     # Metrics to check during canary    metrics:    - name: request-success-rate      thresholdRange:        min: 99  # canary must have > 99% success rate      interval: 1m     - name: request-duration      thresholdRange:        max: 500  # canary p99 latency < 500ms      interval: 30s     - name: error-rate      thresholdRange:        max: 1  # canary error rate < 1%      interval: 1m   # New image version (triggers canary when changed)  template:    metadata:      labels:        app: api    spec:      containers:      - name: api        image: registry.cleanstart.com/cleanstart/python:3.12.1        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: 1000m            memory: 512Mi

How it works: When you update the image in the Deployment, Flagger creates a canary ReplicaSet with the new image and initially routes 0% traffic to it. The system then monitors metrics for one minute. If metrics pass validation, traffic increases by 10%. This process repeats until the deployment reaches 100% or metrics fail. If metrics fail at any point, the system automatically rolls back to the previous version.

Example rollout timeline: The canary deployment begins at Time 0:00 with 10% traffic. At Time 1:00, metrics pass and traffic increases to 20%. At Time 2:00, metrics again pass and traffic increases to 30%. This pattern continues until Time 9:00 when metrics pass and traffic reaches 100%, completing the deployment.

Alternatively, if metrics fail at Time 5:00 (showing an error rate of 2.5%), the deployment automatically rolls back to 0% traffic. By Time 5:30, the canary is aborted and the old version continues to serve traffic.

Automated Canary with Argo Rollouts

Argo Rollouts is another popular framework with more sophisticated traffic management.

Installation:

kubectl create namespace argo-rolloutskubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Rollout Configuration:

apiVersion: argoproj.io/v1alpha1kind: Rolloutmetadata:  name: apispec:  replicas: 10   selector:    matchLabels:      app: api   # Strategy for rolling out new version  strategy:    canary:      steps:      - setWeight: 10   # Step 1: 10% to canary      - pause: {duration: 5m}  # Wait 5 minutes      - setWeight: 25   # Step 2: 25% to canary      - pause: {duration: 5m}      - setWeight: 50   # Step 3: 50% to canary      - pause: {duration: 10m}      - setWeight: 75   # Step 4: 75% to canary      - pause: {duration: 5m}      # Step 5: automatic promotion to 100% (no explicit step needed)       # Automatic rollback on metric failures      analysis:        interval: 1m        threshold: 3        metrics:        - name: error-rate          successCriteria: "< 1"        - name: latency-p99          successCriteria: "< 500"   template:    metadata:      labels:        app: api    spec:      containers:      - name: api        image: registry.cleanstart.com/cleanstart/python:3.12.1        ports:        - containerPort: 8000        livenessProbe:          httpGet:            path: /health            port: 8000          periodSeconds: 10        readinessProbe:          httpGet:            path: /ready            port: 8000          periodSeconds: 5        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: 1000m            memory: 512Mi

Strategy 2: Blue-Green Deployment

Concept

Maintain two complete, identical production environments: Blue (current) and Green (new). Route all traffic to Blue. Deploy new image version to Green. Once Green passes validation, instantly switch traffic to Green. If problems occur, switch back to Blue instantly.

Blue-green deployments offer several advantages over canary approaches. The instant rollback capability means you can revert all traffic to the previous version in seconds rather than waiting for gradual traffic shifting. Full environment testing with real traffic patterns becomes possible because the entire Green environment runs production workload before switching. The strategy is simpler to understand and operate because you're simply switching between two complete environments rather than managing percentages. Blue-green deployment is especially valuable for large image changes with high risk.

The disadvantages include higher resource usage because you maintain 2x infrastructure running simultaneously. A larger blast radius exists if you switch incorrectly because all users are affected instantly. You have less observability during the transition because there's no gradual signal—you either switch all traffic or none.

Blue-Green with Kubernetes

---# Blue deployment (current stable version)apiVersion: apps/v1kind: Deploymentmetadata:  name: api-blue  labels:    app: api    version: bluespec:  replicas: 5  selector:    matchLabels:      app: api      version: blue  template:    metadata:      labels:        app: api        version: blue    spec:      containers:      - name: api        image: registry.cleanstart.com/cleanstart/python:3.11.5        ports:        - containerPort: 8000        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: 1000m            memory: 512Mi ---# Green deployment (new version, not yet receiving traffic)apiVersion: apps/v1kind: Deploymentmetadata:  name: api-green  labels:    app: api    version: greenspec:  replicas: 5  selector:    matchLabels:      app: api      version: green  template:    metadata:      labels:        app: api        version: green    spec:      containers:      - name: api        image: registry.cleanstart.com/cleanstart/python:3.12.1        ports:        - containerPort: 8000        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: 1000m            memory: 512Mi ---# Service routes to blue initiallyapiVersion: v1kind: Servicemetadata:  name: apispec:  selector:    app: api    version: blue  # Traffic goes to blue  ports:  - protocol: TCP    port: 80    targetPort: 8000  type: LoadBalancer

Switchover Procedure:

1. Prepare Green: Ensure Green deployment is fully healthy by checking that all pods are Running and Ready. Use kubectl to verify the pod status before proceeding with traffic switching. kubectl get pods -l version=green # All pods should be Running and Ready
2. Smoke Test Green: Send a small percentage of traffic to Green to verify basic functionality. Create a separate Service for testing that points to the Green deployment. # Create a separate Service for testing kubectl create service clusterip api-green-test \ --tcp=80:8000 \ -o yaml | kubectl apply -f - # Update selector to point to green kubectl patch service api-green-test -p '{"spec":{"selector":{"version":"green"}}}' # Test: curl http://api-green-test
3. Switch Traffic to Green: Update the Service selector to point to Green, instantly directing all traffic to the new version. kubectl patch service api -p '{"spec":{"selector":{"version":"green"}}}'
4. Monitor Green: Watch metrics closely for 10–15 minutes to ensure the deployment remains healthy under full production load. Track error rates, latency, and resource usage. # Watch error rates, latency, resource usage watch 'kubectl top pods -l version=green'
5. Rollback if Needed: If problems emerge, instantly revert traffic back to Blue with a single kubectl command. kubectl patch service api -p '{"spec":{"selector":{"version":"blue"}}}'
6. Cleanup: Once Green has been stable for 24+ hours, delete the Blue deployment to free resources. kubectl delete deployment api-blue

Blue-Green with Istio

If using Istio service mesh for advanced traffic management:

---apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata:  name: apispec:  hosts:  - api  http:  - route:    - destination:        host: api        port:          number: 80        subset: blue  # Routes to blue      weight: 100    - destination:        host: api        port:          number: 80        subset: green  # Not used initially      weight: 0 ---apiVersion: networking.istio.io/v1beta1kind: DestinationRulemetadata:  name: apispec:  host: api  trafficPolicy:    connectionPool:      tcp:        maxConnections: 1000      http:        http1MaxPendingRequests: 1000        http2MaxRequests: 1000  subsets:  - name: blue    labels:      version: blue  - name: green    labels:      version: green

Switchover with Istio: To test Green with partial traffic, modify the VirtualService to route 50% traffic to Green and 50% to Blue, monitoring metrics throughout. Once you confirm Green is healthy, switch 100% traffic to Green:

# Switch 50% traffic to green (for testing)kubectl patch vs api --type merge -p '{  "spec": {    "http": [      {        "route": [          {"destination": {"host": "api", "subset": "blue"}, "weight": 50},          {"destination": {"host": "api", "subset": "green"}, "weight": 50}        ]      }    ]  }}' # Switch 100% traffic to greenkubectl patch vs api --type merge -p '{  "spec": {    "http": [      {        "route": [          {"destination": {"host": "api", "subset": "green"}, "weight": 100}        ]      }    ]  }}'

Strategy 3: Progressive Delivery with Feature Flags

When you want even more control, use feature flags as an additional dimension alongside image versions.

apiVersion: feature.openfeature.dev/v1beta1kind: FeatureFlagmetadata:  name: image-version-python-3.12spec:  rules:  - name: use-new-version    condition: '`user.imageVersionTier` == "early-adopter"'    variants:    - value: "true"      weight: 100  - name: use-stable-version    condition: 'default'    variants:    - value: "false"      weight: 100

This approach allows you to segment users so that internal staff get the new version first before general availability. You can A/B test performance differences between image versions. You can quickly disable the new version for specific user cohorts if problems emerge. Feature flags combine powerfully with canary deployments—the canary deployment handles traffic shifting while feature flags provide additional user-level control.

Monitoring During Rollout

Critical Metrics to Watch

Application-level metrics that indicate service health include error rate (tracking 5xx, 4xx, and application-specific errors), request latency (monitoring p50, p95, and p99 percentiles), request throughput (measuring requests per second), and cache hit rates when applicable.

Infrastructure metrics that signal operational problems include CPU usage per pod, memory usage per pod, disk I/O throughput (read and write operations), and network throughput (incoming and outgoing bytes).

Business metrics that indicate customer impact include transaction completion rate, customer-facing error rate, and revenue impact when applicable.

Prometheus Queries for Canary Monitoring

# Error rate: percentage of failed requestsrate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100 # Latency p99histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) # Memory usage growthcontainer_memory_usage_bytes{pod=~"api-canary.*"} # CPU usagerate(container_cpu_usage_seconds_total{pod=~"api-canary.*"}[5m]) # Requests by versionsum by (version) (rate(http_requests_total[5m])) # Error rate by version(sum by (version) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (version) (rate(http_requests_total[5m]))) * 100

Grafana Dashboard

Create a dashboard with these panels:

{  "dashboard": {    "title": "Canary Rollout Monitor",    "panels": [      {        "title": "Error Rate by Version",        "targets": [          {            "expr": "(sum by (version) (rate(http_requests_total{status=~\"5..\"}[5m])) / sum by (version) (rate(http_requests_total[5m]))) * 100"          }        ]      },      {        "title": "Latency P99 by Version",        "targets": [          {            "expr": "histogram_quantile(0.99, sum by (version, le) (rate(http_request_duration_seconds_bucket[5m])))"          }        ]      },      {        "title": "Memory Usage by Version",        "targets": [          {            "expr": "avg by (version) (container_memory_usage_bytes)"          }        ]      },      {        "title": "Traffic Distribution",        "targets": [          {            "expr": "sum by (version) (rate(http_requests_total[5m]))"          }        ]      }    ]  }}

Automated Rollback Triggers

Define conditions that trigger automatic rollback:

apiVersion: flagger.app/v1beta1kind: Canarymetadata:  name: apispec:  analysis:    # Automatic rollback if any check fails    threshold: 1     metrics:    # Error rate > 1% above baseline = rollback    - name: request-success-rate      thresholdRange:        min: 99      interval: 1m      failureLimit: 3     # Latency p99 > 2x baseline = rollback    - name: request-duration      thresholdRange:        max: 1000  # Assumes baseline is ~500ms      interval: 1m      failureLimit: 2     # Memory growth > 50% = rollback    - name: memory-utilization      thresholdRange:        max: 150      interval: 2m      failureLimit: 2     # Any container restart = immediate rollback    - name: pod-restart-rate      thresholdRange:        max: 0      interval: 1m      failureLimit: 1

Post-Deploy Security Scanning

Automatically scan new image for new vulnerabilities:

apiVersion: batch/v1kind: Jobmetadata:  name: scan-canary-imagespec:  template:    spec:      containers:      - name: scanner        image: aquasec/trivy:latest        command:        - sh        - -c        - |          trivy image --severity HIGH,CRITICAL \            registry.cleanstart.com/cleanstart/python:3.12.1           # If critical vulns found, fail the job          # This triggers canary rollback          exit $?       restartPolicy: Never

Multi-Service Upgrade Coordination

When upgrading multiple CleanStart images simultaneously (e.g., Python service, Go service, Node service):

1. Dependency Analysis

Map service dependencies to understand which services depend on which:

api (Python 3.12) → databasecache (Go 1.22) → apifrontend (Node 22) → api, cache

2. Upgrade Order

Always upgrade in dependency order starting with lowest-level services first (database, cache), then moving to middle-tier services (api), and finally upgrading high-level services (frontend). The rationale is simple: if a cache upgrade breaks the api service, you catch it immediately rather than discovering it later when the frontend depends on a broken api.

3. Staged Rollout

Implement a staged approach across multiple days. On Day 1, upgrade database and cache services with canary. On Day 2, monitor stability without changing anything. On Day 3, upgrade the api service with canary. On Day 4, monitor and verify api compatibility with new cache and database. On Day 5, upgrade frontend service with canary.

4. Rollback Playbook

If any service fails during upgrade, immediately rollback that specific service to the previous version. Keep other services on the new version because they're compatible. Perform root cause analysis to understand what failed. Fix the issue (perhaps a configuration change or code patch is needed). Re-attempt the upgrade with canary once the fix is ready.

Rollout Timeline Template

Use this as a template for your specific service:

## Image Upgrade Plan: Python 3.11 → 3.12 **Planned Window**: 2025-03-24 to 2025-03-28 **Service SLO**: 99.9% availability, < 500ms p99 latency ### Day 0: Preparation (March 24)- [ ] Create canary and stable deployments- [ ] Prepare monitoring dashboards- [ ] Test pre-pulling images to nodes- [ ] Verify internal registry has new image- [ ] Prepare rollback procedure- [ ] Notify on-call team ### Day 1: Canary (March 25)- [ ] Deploy canary with 10% traffic- [ ] Monitor error rate, latency, resource usage for 4 hours- [ ] If stable: progress to Day 2- [ ] If problems: rollback immediately **Expected timeline**:- 08:00 UTC: Canary deployed (10% traffic)- 09:00 UTC: 25% traffic- 10:00 UTC: 50% traffic- 12:00 UTC: 75% traffic- 14:00 UTC: 100% traffic (done!) ### Day 2: Stability (March 26)- [ ] Monitor for 24 hours with no changes- [ ] Confirm no error rate increase- [ ] Confirm resource usage is stable- [ ] Verify all features work correctly- [ ] Check CloudTrail/audit logs for issues ### Day 3: Decommission (March 27)- [ ] Delete old version deployment- [ ] Clean up unused images from nodes- [ ] Verify cluster stability- [ ] Document any issues encountered ### Day 4: Post-Rollout (March 28)- [ ] Update documentation with new version- [ ] Review metrics and performance- [ ] Post-incident review (if any issues)- [ ] Plan next upgrade **Success criteria**: The deployment should achieve zero production errors during rollout, maintain an error rate within 0.1% of baseline, keep latency within 5% of baseline, maintain memory usage within 10% of baseline, and result in no customer complaints or escalations.

Troubleshooting Rollout Issues

Canary Error Rate High

Symptoms: Canary pods seeing > 1% error rate

Diagnosis: Check pod logs to understand what errors are occurring and compare them to stable version logs. Check pod events to see if there are resource or scheduling issues. Check whether the canary pods have sufficient resources. Look at downstream service compatibility.

# Check pod logskubectl logs -l version=canary --tail=50 # Check eventskubectl describe pod -l version=canary # Compare to stable logskubectl logs -l version=stable --tail=50 # Check if it's a resource issuekubectl top pods -l version=canary

Solutions: Examine application logs for specific errors. If resource-constrained, increase limits in canary. If dependency issue, verify downstream services are compatible. If transient, wait longer and check if error rate stabilizes. If persistent, rollback immediately and investigate root cause.

Canary Memory Leak

Symptoms: Memory usage grows during canary period, never stabilizes

Diagnosis: Monitor memory over time to track whether it continuously grows. Check for growing goroutines or threads in the application.

# Monitor memory over timewatch 'kubectl top pods -l version=canary' # Check for growing goroutines/threadskubectl exec pod-name -- /proc/*/status | grep Threads

Solutions: Check if application has a known memory leak in this version. Check if CleanStart updated a memory-heavy dependency. Check if your workload has changed (more requests means more memory). Rollback if leak is severe. Investigate if this is expected behavior for new version.

Canary Latency Increase

Symptoms: Canary seeing 2x baseline latency

Diagnosis: Check if CPU is the bottleneck by examining CPU usage percentages. Check disk I/O to see if the issue is storage-related. Check garbage collection behavior for Go and Java applications.

# Check if it's CPU-boundkubectl top pods -l version=canary  # Check CPU % # Check if it's I/O-boundiostat -x 1 # Check if it's due to garbage collection# (for Go/Java applications)kubectl logs pod-name | grep "GC pause"

Solutions: If CPU-bound, increase CPU limits or reduce traffic to canary. If I/O-bound, check disk health and verify downstream database latency. If GC-bound, this may be expected with new version (tune GC flags if possible). If temporary, wait for warmup period (JIT compilation, cache population).

Canary Stuck at 50%

Symptoms: Canary progresses to 50% but metric check fails at next step

Diagnosis: Flagger is seeing consistent metric failure. Check the canary status for analysis failure reason.

# Check canary statuskubectl describe canary api

Solutions: Adjust metric thresholds if they're too strict. Wait longer since sometimes 50/50 load causes transient issues. Manually increase percentage if needed.

What to Read Next

To deepen your understanding of canary deployments and related production operations, consult 06-operate/kubernetes-helm/supply-chain-dr-plan.md for guidance on preparing for CleanStart registry outages with image mirroring. Review 06-operate/observability/alerting-playbooks.md to set up alerts specific to image upgrade rollouts. Additionally, read 08-reference/deployment/container-image-security.md to understand how to verify image signatures before rolling out new versions.