Knowledge Hub

Air-Gapped Deployment: Disconnected Kubernetes Clusters

Why Air-Gapped Deployments Matter

Air-gapped environments have zero internet connectivity, making them necessary for highly regulated and security-sensitive organizations. Federal and Defense contractors operating on classified networks like SIPR and JWICS require air-gapped deployments for security clearance compliance. Healthcare organizations operating under HIPAA requirements often deploy air-gapped systems to eliminate any risk of data exfiltration across network boundaries. Financial services firms maintain isolated trading floors and payment processing systems using air-gapped architectures to prevent unauthorized network access. Critical infrastructure organizations managing power grids, water systems, and other essential services deploy air-gapped systems to protect against remote attacks.

In air-gapped environments, you cannot pull container images directly from public registries, fetch Helm charts from the internet, or retrieve security signatures from online sources. All of these artifacts must be pre-staged within your private registry and network before deployment, requiring a deliberate preparation workflow before any applications can be deployed to air-gapped clusters.

Registry Mirroring Strategy

The foundation of air-gapped deployment is to copy images from the internet to your private registry, then deploy offline.

Prerequisites

On your internet-connected machine (preparation environment):

# Install toolsbrew install crane skopeo # Authenticate to source registriesgcloud auth application-default login  # For GCRaws ecr get-login-password | docker login --username AWS --password-stdin $AWS_ACCOUNT.dkr.ecr.$REGION.amazonaws.comdocker login -u $DOCKER_USER  # For Docker Hub

Using `crane` (Recommended)

crane is the fastest and most reliable tool:

# Copy single imagecrane copy gcr.io/cleanstart/python:3.11-prod \  private.registry.local/cleanstart/python:3.11-prod # Copy multiple images (batch)cat << 'EOF' > images-to-mirror.txtgcr.io/cleanstart/python:3.11-prodgcr.io/cleanstart/node:20-prodgcr.io/cleanstart/java:21-prodgcr.io/cleanstart/postgresql:16-prodgcr.io/cleanstart/nginx:latestgcr.io/cleanstart/redis:7.2-prodEOF # Parallel copy (fast)cat images-to-mirror.txt | xargs -P 4 -I {} crane copy {} \  $(echo {} | sed 's|gcr.io/cleanstart|private.registry.local/cleanstart|g') # Verify copycrane digest private.registry.local/cleanstart/python:3.11-prod

Using `skopeo` (Alternative)

If crane doesn't work with your registry:

# Copy with TLS verification disabled (for self-signed certs)skopeo copy \  --src-creds=user:password \  --dest-creds=user:password \  --src-tls-verify=false \  --dest-tls-verify=false \  docker://gcr.io/cleanstart/python:3.11-prod \  docker://private.registry.local/cleanstart/python:3.11-prod # Batch copy with retriesfor img in $(cat images-to-mirror.txt); do  skopeo copy \    --src-creds=$GCR_USER:$GCR_PASS \    --dest-creds=$REGISTRY_USER:$REGISTRY_PASS \    --dest-tls-verify=false \    "docker://$img" \    "docker://$(echo $img | sed 's|gcr.io/cleanstart|private.registry.local/cleanstart|g')" \  || echo "Failed: $img"done

Using `docker` Save/Load (Last Resort)

For registries that don't support remote API:

# Pull imagedocker pull gcr.io/cleanstart/python:3.11-prod # Save to tarballdocker save gcr.io/cleanstart/python:3.11-prod \  > python-3.11-prod.tar # Transfer tarball to air-gapped network (via USB, physical media, etc.)scp python-3.11-prod.tar air-gapped-server:/tmp/ # On air-gapped machine:docker load < /tmp/python-3.11-prod.tardocker tag cleanstart/python:3.11-prod \  private.registry.local/cleanstart/python:3.11-proddocker push private.registry.local/cleanstart/python:3.11-prod # Compress for transfertar -czf cleanstart-images.tar.gz *.tar# Transfer single 2-5 GB file instead of many small ones

Helm Chart Mirroring

Mirror Helm charts used by CleanStart deployments.

Download Charts from Internet

# Add CleanStart Helm repohelm repo add cleanstart https://charts.cleanstart.devhelm repo update # Download charts (as tarballs)helm pull cleanstart/cleanstart-core --version 2.0.0helm pull cleanstart/cleanstart-database --version 2.0.0helm pull cleanstart/cleanstart-observability --version 2.0.0 # List what you downloadedls -lh cleanstart-*.tgz

Upload to Private Chart Repository

Option A: Using ChartMuseum

# On internet machine, push to ChartMuseumcurl --data-binary "@cleanstart-core-2.0.0.tgz" \  http://chartmuseum.private.registry.local/api/charts # On air-gapped machine, add repohelm repo add cleanstart \  http://chartmuseum.private.registry.local \  --username admin --password $CHARTMUSEUM_PASSWORD helm install my-release cleanstart/cleanstart-core \  --version 2.0.0 \  --values values.yaml

Option B: Using Artifactory

# Push from internet machinecurl -H "X-JFrog-Art-Api: $ARTIFACTORY_TOKEN" \  -T cleanstart-core-2.0.0.tgz \  "https://artifactory.private.registry.local/artifactory/helm-local/" # Add to air-gapped helmhelm repo add cleanstart \  https://artifactory.private.registry.local/artifactory/helm/ \  --username admin \  --password $ARTIFACTORY_PASSWORD helm install my-release cleanstart/cleanstart-core

Option C: Offline Bundle (No HTTP)

For fully disconnected environments, ship charts as tarballs:

# On internet machine:mkdir -p air-gap-bundle/chartscp cleanstart-*.tgz air-gap-bundle/charts/ tar -czf air-gap-deployment-bundle.tar.gz air-gap-bundle/ # Transfer bundle via USB/physical media to air-gapped environment # On air-gapped machine:tar -xzf air-gap-deployment-bundle.tar.gz # Install chart locallyhelm install my-release ./air-gap-bundle/charts/cleanstart-core-2.0.0.tgz \  --values values.yaml

Artifact Mirroring (SBOMs, Signatures, Attestations)

Copy Signatures (Cosign)

Copy image signatures used for verification:

# On internet machine:# Export public key for signature verificationcosign public-key --key $COSIGN_KEY > cosign.pub # Copy image + signaturecrane copy gcr.io/cleanstart/python:3.11-prod \  private.registry.local/cleanstart/python:3.11-prod \  --preserve-image-references # Cosign stores signatures as separate OCI artifacts# crane copy automatically handles them # Verify on air-gapped machine (after importing public key)cosign verify \  --key cosign.pub \  private.registry.local/cleanstart/python:3.11-prod

Mirror SBOMs (Software Bill of Materials)

SBOMs are JSON/XML files describing what's in an image:

# On internet machine:# SBOM is usually stored as OCI artifact alongside imagesyft gcr.io/cleanstart/python:3.11-prod -o spdx-json > python-3.11.sbom.spdx.json # Or fetch via OCI registry APIcurl -s https://gcr.io/v2/cleanstart/python/manifests/3.11-prod \  -H "Accept: application/vnd.oci.image.manifest.v1+json" \  | jq '.annotations' | grep sbom # Copy SBOM artifactoras copy \  gcr.io/cleanstart/python:3.11-prod:sbom.spdx \  private.registry.local/cleanstart/python:3.11-prod:sbom.spdx

Copy Attestations

Copy supply chain attestations (Sigstore, in-toto):

# On internet machine:# Attestations stored in OCI registrycosign tree gcr.io/cleanstart/python:3.11-prod # Copy attestations alongside imagecrane copy gcr.io/cleanstart/python:3.11-prod \  private.registry.local/cleanstart/python:3.11-prod # Then verify on air-gapped machinecosign verify-attestation \  --key cosign.pub \  --type slsaprovenance \  private.registry.local/cleanstart/python:3.11-prod

Private Registry Setup

Option A: Harbor (Recommended)

Harbor is the industry standard for air-gapped registries:

# Install on air-gapped Kubernetes clusterhelm repo add harbor https://charts.goharbor.iohelm pull harbor/harbor --version 1.13.0 helm install harbor ./harbor-1.13.0.tgz \  --namespace harbor \  --create-namespace \  --values - <<EOFexpose:  type: nodePort  nodePort:    http:      nodePort: 30080    https:      nodePort: 30443externalURL: https://harbor.local.network:30443 persistence:  enabled: true  resourceClass: "fast-ssd" harborAdminPassword: SuperSecure@2024 portal:  replicas: 2core:  replicas: 2jobservice:  replicas: 2database:  type: internal  internal:    password: DbSecure@2024redis:  type: internalEOF # Configure image pull secretkubectl create secret docker-registry harbor-secret \  --docker-server=harbor.local.network:30443 \  --docker-username=admin \  --docker-password=SuperSecure@2024 \  --docker-email=admin@harbor.local.network \  --namespace=production

Option B: Artifactory

Artifactory is good if you already use it:

# Configure in values.yamlhelm install artifactory jfrog/artifactory \  --values - <<EOFartifactory:  licenses:    - $LICENSE_KEY  image:    repository: releases-docker.jfrog.io/jfrog/artifactory-pro  persistence:    enabled: true    size: 500GisystemYaml: |  configVersion: 2  shared:    security:      encryptionKey: "$ENCRYPTION_KEY"EOF # Create Docker repositorycurl -u admin:password \  -X PUT \  -H "Content-Type: application/json" \  -d '{    "key": "docker-private",    "packageType": "Docker",    "dockerApiVersion": "V2"  }' \  https://artifactory.local.network/artifactory/api/repositories

Option C: MinIO (S3-Compatible)

For cost-sensitive deployments:

helm install minio minio/minio \  --set auth.rootUser=admin \  --set auth.rootPassword=MinioSecure@2024 \  --set persistence.size=500Gi \  --namespace storage # Configure OCI registry on top of MinIO# (requires additional setup with OCI registry bridge)

Signature Verification in Air-Gapped

Once your public key is imported, verify images offline:

# 1. Export public key (on internet machine)cosign public-key --key gcpkms://projects/my-project/locations/us/keyRings/my-ring/cryptoKeys/my-key \  > cosign.pub # 2. Transfer cosign.pub to air-gapped environment (USB, email, etc.) # 3. On air-gapped machine, verify imagescosign verify \  --key cosign.pub \  private.registry.local/cleanstart/python:3.11-prod # 4. Store public key in ConfigMap for cluster accesskubectl create configmap cosign-keys \  --from-file=cosign.pub \  --namespace=security \  --dry-run=client \  -o yaml | kubectl apply -f - # 5. Configure admission controller to enforce signature verification# See CleanStart security policy docs

CI/CD in Air-Gapped Networks

GitLab Runner (On-Premises)

# gitlab-runner-values.yamlgitlabUrl: https://gitlab.local.network/gitlabToken: $GITLAB_RUNNER_TOKEN runners:  config: |    [[runners]]      [runners.kubernetes]        image = "private.registry.local/cleanstart/ubuntu:latest"        privileged = true        [[runners.kubernetes.volumes.secret]]          name = "registry-credentials"          mount_path = "/root/.docker" rbac:  create: true  serviceAccountName: gitlab-runner

Deploy:

helm repo add gitlab https://charts.gitlab.iohelm install gitlab-runner gitlab/gitlab-runner \  --namespace gitlab-runner \  --create-namespace \  --values gitlab-runner-values.yaml # Configure CI/CD pipeline to use private registrycat > .gitlab-ci.yml <<EOFstages:  - build  - deploy build:app:  stage: build  image: private.registry.local/cleanstart/golang:1.21-dev  script:    - go build -o app .    - crane push app:latest private.registry.local/app:$(git rev-parse --short HEAD)  after_script:    - cosign sign --key /secret/cosign.key \        private.registry.local/app:$(git rev-parse --short HEAD) deploy:prod:  stage: deploy  image: private.registry.local/cleanstart/kubectl:latest  script:    - kubectl set image deployment/app \        app=private.registry.local/app:$(git rev-parse --short HEAD)EOF

Jenkins On-Premises

// Jenkinsfile for air-gappedpipeline {  agent any   environment {    REGISTRY = 'private.registry.local'    IMAGE = "${REGISTRY}/app"    DOCKER_CONFIG = '/var/jenkins_home/.docker'  }   stages {    stage('Build') {      steps {        sh '''          docker build -t ${IMAGE}:${BUILD_NUMBER} .          docker push ${IMAGE}:${BUILD_NUMBER}           # Sign image          cosign sign --key /var/jenkins_home/cosign.key \            ${IMAGE}:${BUILD_NUMBER}        '''      }    }     stage('Deploy') {      steps {        sh '''          kubectl set image deployment/app \            app=${IMAGE}:${BUILD_NUMBER} \            --record \            -n production           # Verify deployment          kubectl rollout status deployment/app -n production        '''      }    }  }}

Update Workflow (Periodic Sync)

Set up automated sync from CleanStart to your private registry:

#!/bin/bash# sync-cleanstart-images.sh — runs daily/weekly set -e SYNC_LOG="/var/log/cleanstart-sync.log"TIMESTAMP=$(date +%Y-%m-%d\ %H:%M:%S) # Source registry (CleanStart)SOURCE_REGISTRY="gcr.io/cleanstart"DEST_REGISTRY="private.registry.local/cleanstart" # List of images to sync (update as needed)IMAGES=(  "python:3.11-prod"  "node:20-prod"  "java:21-prod"  "postgresql:16-prod"  "redis:7.2-prod"  "nginx:latest") echo "[$TIMESTAMP] Starting CleanStart image sync" | tee -a $SYNC_LOG for img in "${IMAGES[@]}"; do  echo "[$TIMESTAMP] Syncing $SOURCE_REGISTRY/$img..." | tee -a $SYNC_LOG   if crane copy "$SOURCE_REGISTRY/$img" "$DEST_REGISTRY/$img" 2>>$SYNC_LOG; then    echo "[$TIMESTAMP] ✅ Synced: $img" | tee -a $SYNC_LOG  else    echo "[$TIMESTAMP] ❌ Failed: $img" | tee -a $SYNC_LOG    # Alert ops team    mail -s "CleanStart sync failed: $img" ops@company.local < $SYNC_LOG  fidone echo "[$TIMESTAMP] Sync complete" | tee -a $SYNC_LOG

Schedule with cron:

# Run sync every Sunday at 2 AM0 2 * * 0 /usr/local/bin/sync-cleanstart-images.sh

Or use Kubernetes CronJob:

apiVersion: batch/v1kind: CronJobmetadata:  name: cleanstart-sync  namespace: kube-systemspec:  schedule: "0 2 * * 0"  # 2 AM Sundays  jobTemplate:    spec:      template:        spec:          serviceAccountName: registry-sync          containers:          - name: syncer            image: private.registry.local/cleanstart/golang:1.21-prod            command:            - /bin/sh            - -c            - |              /scripts/sync-cleanstart-images.sh            volumeMounts:            - name: sync-script              mountPath: /scripts            - name: registry-creds              mountPath: /root/.docker          volumes:          - name: sync-script            configMap:              name: sync-script              defaultMode: 0755          - name: registry-creds            secret:              secretName: registry-credentials          restartPolicy: OnFailure

Troubleshooting Air-Gapped Deployments

Image Pull Fails

# Verify image exists in private registrycurl -s -u admin:password \  https://private.registry.local/v2/cleanstart/python/manifests/3.11-prod # Check imagePullSecret is mountedkubectl describe pod app-123abc -n production | grep -A 5 "Pull Secrets" # Verify secret exists and has credentialskubectl get secret registry-credentials -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq . # Test pull manually from podkubectl exec -it pod/app-123abc -n production -- \  docker pull private.registry.local/cleanstart/python:3.11-prod

Signature Verification Fails

# Verify public key is correctcosign public-key --key cosign-key.pem | openssl rsa -text # Check image was actually signedcosign tree private.registry.local/cleanstart/python:3.11-prod # Verify with verbose outputcosign verify --key cosign.pub --verbose \  private.registry.local/cleanstart/python:3.11-prod

Network Connectivity Issues

# Test registry connectivity from podkubectl run test-pod \  --image=private.registry.local/cleanstart/curl:latest \  -it --rm \  -- curl -I https://private.registry.local/v2/ # Check DNS from podkubectl run test-pod \  --image=private.registry.local/cleanstart/alpine:latest \  -it --rm \  -- nslookup private.registry.local

Security Checklist

Before deploying to air-gapped environment:

All images mirrored to private registry. All Helm charts mirrored (no internet access). Public key for signatures imported and secured. Private registry has TLS certificate (self-signed OK). Image pull secrets created for all namespaces. Signature verification enabled on admission controller. Registry backup configured (critical system!). Network policies block unnecessary egress. Registry admin passwords changed from defaults. Sync process tested and scheduled.