Deep Dive: Deploying Kubernetes in Air-Gapped Networks — The Five Things That Break First

I've watched three different teams try to stand up Kubernetes behind an air gap, and every single one hit the same five walls in the same order. Not because they didn't know Kubernetes — because air-gapped networking breaks assumptions that most Kubernetes tooling doesn't even know it's making.

None of these five things are exotic. They're the parts of a cluster that quietly assume the internet is one hop away: the registry, the certificate chain, the chart repos, the bootstrap artifacts, and the upgrade cadence. Here's where each one breaks, and what actually fixes it.

1. The container registry is the first wall, and it hits before the cluster even exists

Every kubeadm init, every Helm chart, every Deployment manifest assumes it can reach registry.k8s.io, docker.io, or ghcr.io to pull an image. On an air-gapped network there's no path out, so the first pull just hangs or times out — no useful error, just a pod stuck in ImagePullBackOff.

The fix is a pull-through or pre-seeded internal registry that mirrors exactly what you need, by digest, not by floating tag. "Mirror everything" doesn't work at air-gapped scale — you mirror what your manifests actually reference, pinned.

# On a machine WITH internet access: pull, tag for your internal registry, push
IMAGES=(
  "registry.k8s.io/kube-apiserver:v1.31.2"
  "registry.k8s.io/kube-controller-manager:v1.31.2"
  "registry.k8s.io/kube-scheduler:v1.31.2"
  "registry.k8s.io/kube-proxy:v1.31.2"
  "registry.k8s.io/pause:3.10"
  "registry.k8s.io/etcd:3.5.16-0"
  "registry.k8s.io/coredns/coredns:v1.11.3"
)

for IMG in "${IMAGES[@]}"; do
  docker pull "$IMG"
  NEW_TAG="registry.internal.example.com/${IMG#*/}"
  docker tag "$IMG" "$NEW_TAG"
  docker push "$NEW_TAG"
done

Then point every node's container runtime at the mirror instead of the public registry. For containerd (the default on most modern clusters), that's a registry config block, not an environment variable:

# /etc/containerd/certs.d/registry.k8s.io/hosts.toml
server = "https://registry.internal.example.com"

[host."https://registry.internal.example.com"]
  capabilities = ["pull", "resolve"]
  skip_verify = false

Do this for every registry host your manifests reference — registry.k8s.io, docker.io, quay.io, ghcr.io — each gets its own hosts.toml pointing at the same internal mirror, or at separate mirrored paths if you're namespacing them. Skip this and kubeadm init itself fails before you ever get a control plane.

2. TLS trust breaks next, and it's invisible until something tries to call out

Certificates aren't just a kubelet problem. Any component that validates a cert chain against a public CA — an admission webhook calling an external service, an image signature verifier checking Sigstore's public transparency log, cert-manager trying to reach Let's Encrypt — will fail its OCSP or CRL check, because that check is itself a network call to the outside world.

The fix has two parts. First, run your own internal CA and distribute its root cert to every node's trust store, not just to Kubernetes Secrets:

# On every node
cp internal-ca.crt /usr/local/share/ca-certificates/internal-ca.crt
update-ca-certificates

Second — and this is the one teams miss — disable or point OCSP/CRL checking at an internal responder for anything that does certificate revocation checking, and make sure NTP is pointed at an internal time source. Certificate validation fails silently and confusingly if the system clock drifts, because a cert that "isn't valid yet" or "has expired" looks identical to a cert that's just being checked against the wrong clock.

# chrony.conf on an internal NTP source, pointed at by every node
server ntp.internal.example.com iburst
# no public pool.ntp.org entries — they'll just time out

3. Helm chart repositories assume helm repo add can reach the internet

This one surprises people who've already solved the registry problem: helm repo add bitnami https://charts.bitnami.com/bitnami is itself an outbound HTTP call, completely separate from the container images the chart eventually pulls. Solve the registry mirror and you've solved nothing here.

Run an internal chart repository — ChartMuseum or an OCI-compliant registry (Harbor and most modern registries speak OCI for Helm charts now, which means one mirror can serve both images and charts):

# Pull a chart and its dependencies on a connected machine
helm pull bitnami/postgresql --version 15.2.3 --untar
helm dependency update ./postgresql

# Push it to your internal OCI registry
helm package ./postgresql
helm push postgresql-15.2.3.tgz oci://registry.internal.example.com/charts

# Inside the air gap
helm install my-pg oci://registry.internal.example.com/charts/postgresql --version 15.2.3

Watch subcharts specifically — a parent chart that looks self-contained often has a dependencies: block in its Chart.yaml pulling in three more charts from three more repos, each needing its own mirror entry.

4. The cluster bootstrap itself needs artifacts nobody thinks of as "images"

Past the control plane images, kubeadm and your CNI plugin both expect to fetch things that aren't container images at all: CNI binaries (often a GitHub release tarball), the crictl and kubeadm/kubelet binaries themselves on new nodes, and in some setups, sandbox image digests resolved dynamically rather than pinned in your config.

Pin every one of these explicitly in your kubeadm config rather than letting defaults resolve against the public internet:

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
imageRepository: registry.internal.example.com
kubernetesVersion: v1.31.2

Then stage the CNI plugin binaries and the kubelet/kubeadm binaries on an internal file server or artifact repo ahead of time, and reference that internal location in whatever provisioning tooling (Ansible, cloud-init, a golden image) builds your nodes — not a curl https://github.com/... baked into a setup script.

5. The cluster that works today quietly falls behind without a plan to feed it updates

The first four problems get solved once, during bring-up, and teams declare victory. The fifth one is ongoing: every CVE patch, every Kubernetes minor version, every chart update is a repeat of problems 1–4, on a recurring schedule, and air-gapped teams who don't plan for it end up frozen on whatever version they first got working — which is itself a security posture problem a year later.

Build the mirror-and-promote workflow as a repeatable pipeline from day one, not a one-off script: a connected "staging" environment that pulls and validates new image/chart versions, a controlled transfer process (physical media, a one-way data diode, or a scheduled sync window, depending on your environment's actual isolation level), and a promotion step into the internal registry that's identical every time. Teams that treat air-gapped delivery as a pipeline stay current. Teams that treat it as a one-time migration don't.

None of these five are hard problems individually. What makes air-gapped Kubernetes hard is that they all surface in the first week, usually in this order, and most teams only find out about number three or four after they thought they were done.

Mumuni Mohammed

Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.