Why Air-Gapped Deployments Matter
Air-gapped environments have zero internet connectivity, making them necessary for highly regulated and security-sensitive organizations. Federal and Defense contractors operating on classified networks like SIPR and JWICS require air-gapped deployments for security clearance compliance. Healthcare organizations operating under HIPAA requirements often deploy air-gapped systems to eliminate any risk of data exfiltration across network boundaries. Financial services firms maintain isolated trading floors and payment processing systems using air-gapped architectures to prevent unauthorized network access. Critical infrastructure organizations managing power grids, water systems, and other essential services deploy air-gapped systems to protect against remote attacks.
In air-gapped environments, you cannot pull container images directly from public registries, fetch Helm charts from the internet, or retrieve security signatures from online sources. All of these artifacts must be pre-staged within your private registry and network before deployment, requiring a deliberate preparation workflow before any applications can be deployed to air-gapped clusters.
Registry Mirroring Strategy
The foundation of air-gapped deployment is to copy images from the internet to your private registry, then deploy offline.
Prerequisites
On your internet-connected machine (preparation environment):
# Install toolsbrew install crane skopeo # Authenticate to source registriesgcloud auth application-default login # For GCRaws ecr get-login-password | docker login --username AWS --password-stdin $AWS_ACCOUNT.dkr.ecr.$REGION.amazonaws.comdocker login -u $DOCKER_USER # For Docker HubUsing crane (Recommended)
crane is the fastest and most reliable tool:
# Copy single imagecrane copy gcr.io/cleanstart/python:3.11-prod \ private.registry.local/cleanstart/python:3.11-prod # Copy multiple images (batch)cat << 'EOF' > images-to-mirror.txtgcr.io/cleanstart/python:3.11-prodgcr.io/cleanstart/node:20-prodgcr.io/cleanstart/java:21-prodgcr.io/cleanstart/postgresql:16-prodgcr.io/cleanstart/nginx:latestgcr.io/cleanstart/redis:7.2-prodEOF # Parallel copy (fast)cat images-to-mirror.txt | xargs -P 4 -I {} crane copy {} \ $(echo {} | sed 's|gcr.io/cleanstart|private.registry.local/cleanstart|g') # Verify copycrane digest private.registry.local/cleanstart/python:3.11-prodUsing skopeo (Alternative)
If crane doesn't work with your registry:
# Copy with TLS verification disabled (for self-signed certs)skopeo copy \ --src-creds=user:password \ --dest-creds=user:password \ --src-tls-verify=false \ --dest-tls-verify=false \ docker://gcr.io/cleanstart/python:3.11-prod \ docker://private.registry.local/cleanstart/python:3.11-prod # Batch copy with retriesfor img in $(cat images-to-mirror.txt); do skopeo copy \ --src-creds=$GCR_USER:$GCR_PASS \ --dest-creds=$REGISTRY_USER:$REGISTRY_PASS \ --dest-tls-verify=false \ "docker://$img" \ "docker://$(echo $img | sed 's|gcr.io/cleanstart|private.registry.local/cleanstart|g')" \ || echo "Failed: $img"doneUsing docker Save/Load (Last Resort)
For registries that don't support remote API:
# Pull imagedocker pull gcr.io/cleanstart/python:3.11-prod # Save to tarballdocker save gcr.io/cleanstart/python:3.11-prod \ > python-3.11-prod.tar # Transfer tarball to air-gapped network (via USB, physical media, etc.)scp python-3.11-prod.tar air-gapped-server:/tmp/ # On air-gapped machine:docker load < /tmp/python-3.11-prod.tardocker tag cleanstart/python:3.11-prod \ private.registry.local/cleanstart/python:3.11-proddocker push private.registry.local/cleanstart/python:3.11-prod # Compress for transfertar -czf cleanstart-images.tar.gz *.tar# Transfer single 2-5 GB file instead of many small onesHelm Chart Mirroring
Mirror Helm charts used by CleanStart deployments.
Download Charts from Internet
# Add CleanStart Helm repohelm repo add cleanstart https://charts.cleanstart.devhelm repo update # Download charts (as tarballs)helm pull cleanstart/cleanstart-core --version 2.0.0helm pull cleanstart/cleanstart-database --version 2.0.0helm pull cleanstart/cleanstart-observability --version 2.0.0 # List what you downloadedls -lh cleanstart-*.tgzUpload to Private Chart Repository
Option A: Using ChartMuseum
# On internet machine, push to ChartMuseumcurl --data-binary "@cleanstart-core-2.0.0.tgz" \ http://chartmuseum.private.registry.local/api/charts # On air-gapped machine, add repohelm repo add cleanstart \ http://chartmuseum.private.registry.local \ --username admin --password $CHARTMUSEUM_PASSWORD helm install my-release cleanstart/cleanstart-core \ --version 2.0.0 \ --values values.yamlOption B: Using Artifactory
# Push from internet machinecurl -H "X-JFrog-Art-Api: $ARTIFACTORY_TOKEN" \ -T cleanstart-core-2.0.0.tgz \ "https://artifactory.private.registry.local/artifactory/helm-local/" # Add to air-gapped helmhelm repo add cleanstart \ https://artifactory.private.registry.local/artifactory/helm/ \ --username admin \ --password $ARTIFACTORY_PASSWORD helm install my-release cleanstart/cleanstart-coreOption C: Offline Bundle (No HTTP)
For fully disconnected environments, ship charts as tarballs:
# On internet machine:mkdir -p air-gap-bundle/chartscp cleanstart-*.tgz air-gap-bundle/charts/ tar -czf air-gap-deployment-bundle.tar.gz air-gap-bundle/ # Transfer bundle via USB/physical media to air-gapped environment # On air-gapped machine:tar -xzf air-gap-deployment-bundle.tar.gz # Install chart locallyhelm install my-release ./air-gap-bundle/charts/cleanstart-core-2.0.0.tgz \ --values values.yamlArtifact Mirroring (SBOMs, Signatures, Attestations)
Copy Signatures (Cosign)
Copy image signatures used for verification:
# On internet machine:# Export public key for signature verificationcosign public-key --key $COSIGN_KEY > cosign.pub # Copy image + signaturecrane copy gcr.io/cleanstart/python:3.11-prod \ private.registry.local/cleanstart/python:3.11-prod \ --preserve-image-references # Cosign stores signatures as separate OCI artifacts# crane copy automatically handles them # Verify on air-gapped machine (after importing public key)cosign verify \ --key cosign.pub \ private.registry.local/cleanstart/python:3.11-prodMirror SBOMs (Software Bill of Materials)
SBOMs are JSON/XML files describing what's in an image:
# On internet machine:# SBOM is usually stored as OCI artifact alongside imagesyft gcr.io/cleanstart/python:3.11-prod -o spdx-json > python-3.11.sbom.spdx.json # Or fetch via OCI registry APIcurl -s https://gcr.io/v2/cleanstart/python/manifests/3.11-prod \ -H "Accept: application/vnd.oci.image.manifest.v1+json" \ | jq '.annotations' | grep sbom # Copy SBOM artifactoras copy \ gcr.io/cleanstart/python:3.11-prod:sbom.spdx \ private.registry.local/cleanstart/python:3.11-prod:sbom.spdxCopy Attestations
Copy supply chain attestations (Sigstore, in-toto):
# On internet machine:# Attestations stored in OCI registrycosign tree gcr.io/cleanstart/python:3.11-prod # Copy attestations alongside imagecrane copy gcr.io/cleanstart/python:3.11-prod \ private.registry.local/cleanstart/python:3.11-prod # Then verify on air-gapped machinecosign verify-attestation \ --key cosign.pub \ --type slsaprovenance \ private.registry.local/cleanstart/python:3.11-prodPrivate Registry Setup
Option A: Harbor (Recommended)
Harbor is the industry standard for air-gapped registries:
# Install on air-gapped Kubernetes clusterhelm repo add harbor https://charts.goharbor.iohelm pull harbor/harbor --version 1.13.0 helm install harbor ./harbor-1.13.0.tgz \ --namespace harbor \ --create-namespace \ --values - <<EOFexpose: type: nodePort nodePort: http: nodePort: 30080 https: nodePort: 30443externalURL: https://harbor.local.network:30443 persistence: enabled: true resourceClass: "fast-ssd" harborAdminPassword: SuperSecure@2024 portal: replicas: 2core: replicas: 2jobservice: replicas: 2database: type: internal internal: password: DbSecure@2024redis: type: internalEOF # Configure image pull secretkubectl create secret docker-registry harbor-secret \ --docker-server=harbor.local.network:30443 \ --docker-username=admin \ --docker-password=SuperSecure@2024 \ --docker-email=admin@harbor.local.network \ --namespace=productionOption B: Artifactory
Artifactory is good if you already use it:
# Configure in values.yamlhelm install artifactory jfrog/artifactory \ --values - <<EOFartifactory: licenses: - $LICENSE_KEY image: repository: releases-docker.jfrog.io/jfrog/artifactory-pro persistence: enabled: true size: 500GisystemYaml: | configVersion: 2 shared: security: encryptionKey: "$ENCRYPTION_KEY"EOF # Create Docker repositorycurl -u admin:password \ -X PUT \ -H "Content-Type: application/json" \ -d '{ "key": "docker-private", "packageType": "Docker", "dockerApiVersion": "V2" }' \ https://artifactory.local.network/artifactory/api/repositoriesOption C: MinIO (S3-Compatible)
For cost-sensitive deployments:
helm install minio minio/minio \ --set auth.rootUser=admin \ --set auth.rootPassword=MinioSecure@2024 \ --set persistence.size=500Gi \ --namespace storage # Configure OCI registry on top of MinIO# (requires additional setup with OCI registry bridge)Signature Verification in Air-Gapped
Once your public key is imported, verify images offline:
# 1. Export public key (on internet machine)cosign public-key --key gcpkms://projects/my-project/locations/us/keyRings/my-ring/cryptoKeys/my-key \ > cosign.pub # 2. Transfer cosign.pub to air-gapped environment (USB, email, etc.) # 3. On air-gapped machine, verify imagescosign verify \ --key cosign.pub \ private.registry.local/cleanstart/python:3.11-prod # 4. Store public key in ConfigMap for cluster accesskubectl create configmap cosign-keys \ --from-file=cosign.pub \ --namespace=security \ --dry-run=client \ -o yaml | kubectl apply -f - # 5. Configure admission controller to enforce signature verification# See CleanStart security policy docsCI/CD in Air-Gapped Networks
GitLab Runner (On-Premises)
# gitlab-runner-values.yamlgitlabUrl: https://gitlab.local.network/gitlabToken: $GITLAB_RUNNER_TOKEN runners: config: | [[runners]] [runners.kubernetes] image = "private.registry.local/cleanstart/ubuntu:latest" privileged = true [[runners.kubernetes.volumes.secret]] name = "registry-credentials" mount_path = "/root/.docker" rbac: create: true serviceAccountName: gitlab-runnerDeploy:
helm repo add gitlab https://charts.gitlab.iohelm install gitlab-runner gitlab/gitlab-runner \ --namespace gitlab-runner \ --create-namespace \ --values gitlab-runner-values.yaml # Configure CI/CD pipeline to use private registrycat > .gitlab-ci.yml <<EOFstages: - build - deploy build:app: stage: build image: private.registry.local/cleanstart/golang:1.21-dev script: - go build -o app . - crane push app:latest private.registry.local/app:$(git rev-parse --short HEAD) after_script: - cosign sign --key /secret/cosign.key \ private.registry.local/app:$(git rev-parse --short HEAD) deploy:prod: stage: deploy image: private.registry.local/cleanstart/kubectl:latest script: - kubectl set image deployment/app \ app=private.registry.local/app:$(git rev-parse --short HEAD)EOFJenkins On-Premises
// Jenkinsfile for air-gappedpipeline { agent any environment { REGISTRY = 'private.registry.local' IMAGE = "${REGISTRY}/app" DOCKER_CONFIG = '/var/jenkins_home/.docker' } stages { stage('Build') { steps { sh ''' docker build -t ${IMAGE}:${BUILD_NUMBER} . docker push ${IMAGE}:${BUILD_NUMBER} # Sign image cosign sign --key /var/jenkins_home/cosign.key \ ${IMAGE}:${BUILD_NUMBER} ''' } } stage('Deploy') { steps { sh ''' kubectl set image deployment/app \ app=${IMAGE}:${BUILD_NUMBER} \ --record \ -n production # Verify deployment kubectl rollout status deployment/app -n production ''' } } }}Update Workflow (Periodic Sync)
Set up automated sync from CleanStart to your private registry:
#!/bin/bash# sync-cleanstart-images.sh — runs daily/weekly set -e SYNC_LOG="/var/log/cleanstart-sync.log"TIMESTAMP=$(date +%Y-%m-%d\ %H:%M:%S) # Source registry (CleanStart)SOURCE_REGISTRY="gcr.io/cleanstart"DEST_REGISTRY="private.registry.local/cleanstart" # List of images to sync (update as needed)IMAGES=( "python:3.11-prod" "node:20-prod" "java:21-prod" "postgresql:16-prod" "redis:7.2-prod" "nginx:latest") echo "[$TIMESTAMP] Starting CleanStart image sync" | tee -a $SYNC_LOG for img in "${IMAGES[@]}"; do echo "[$TIMESTAMP] Syncing $SOURCE_REGISTRY/$img..." | tee -a $SYNC_LOG if crane copy "$SOURCE_REGISTRY/$img" "$DEST_REGISTRY/$img" 2>>$SYNC_LOG; then echo "[$TIMESTAMP] ✅ Synced: $img" | tee -a $SYNC_LOG else echo "[$TIMESTAMP] ❌ Failed: $img" | tee -a $SYNC_LOG # Alert ops team mail -s "CleanStart sync failed: $img" ops@company.local < $SYNC_LOG fidone echo "[$TIMESTAMP] Sync complete" | tee -a $SYNC_LOGSchedule with cron:
# Run sync every Sunday at 2 AM0 2 * * 0 /usr/local/bin/sync-cleanstart-images.shOr use Kubernetes CronJob:
apiVersion: batch/v1kind: CronJobmetadata: name: cleanstart-sync namespace: kube-systemspec: schedule: "0 2 * * 0" # 2 AM Sundays jobTemplate: spec: template: spec: serviceAccountName: registry-sync containers: - name: syncer image: private.registry.local/cleanstart/golang:1.21-prod command: - /bin/sh - -c - | /scripts/sync-cleanstart-images.sh volumeMounts: - name: sync-script mountPath: /scripts - name: registry-creds mountPath: /root/.docker volumes: - name: sync-script configMap: name: sync-script defaultMode: 0755 - name: registry-creds secret: secretName: registry-credentials restartPolicy: OnFailureTroubleshooting Air-Gapped Deployments
Image Pull Fails
# Verify image exists in private registrycurl -s -u admin:password \ https://private.registry.local/v2/cleanstart/python/manifests/3.11-prod # Check imagePullSecret is mountedkubectl describe pod app-123abc -n production | grep -A 5 "Pull Secrets" # Verify secret exists and has credentialskubectl get secret registry-credentials -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq . # Test pull manually from podkubectl exec -it pod/app-123abc -n production -- \ docker pull private.registry.local/cleanstart/python:3.11-prodSignature Verification Fails
# Verify public key is correctcosign public-key --key cosign-key.pem | openssl rsa -text # Check image was actually signedcosign tree private.registry.local/cleanstart/python:3.11-prod # Verify with verbose outputcosign verify --key cosign.pub --verbose \ private.registry.local/cleanstart/python:3.11-prodNetwork Connectivity Issues
# Test registry connectivity from podkubectl run test-pod \ --image=private.registry.local/cleanstart/curl:latest \ -it --rm \ -- curl -I https://private.registry.local/v2/ # Check DNS from podkubectl run test-pod \ --image=private.registry.local/cleanstart/alpine:latest \ -it --rm \ -- nslookup private.registry.localSecurity Checklist
Before deploying to air-gapped environment:
All images mirrored to private registry. All Helm charts mirrored (no internet access). Public key for signatures imported and secured. Private registry has TLS certificate (self-signed OK). Image pull secrets created for all namespaces. Signature verification enabled on admission controller. Registry backup configured (critical system!). Network policies block unnecessary egress. Registry admin passwords changed from defaults. Sync process tested and scheduled.