Kubernetes Secrets Are Just Base64, Not Encrypted - Here's How to Fix That

If you store a password in a Kubernetes Secret and think it's encrypted, it probably isn't. By default, a Secret is base64-encoded text sitting in etcd, and base64 is encoding, not encryption — anyone with read access to etcd, or a backup of it, can decode it in one command.

What a Secret actually protects you from, by default

Run this against any Secret in a default cluster setup:

kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d

That gives you the plaintext password, instantly, with no key and no decryption step. A Secret's only default protections over a ConfigMap are: it's not displayed in kubectl get output by default, kubelet avoids writing it to disk unencrypted in some configurations, and RBAC can gate get/list on the secrets resource separately from other objects. None of that protects the data at rest in etcd, and none of it protects an etcd backup or snapshot that ends up somewhere it shouldn't.

This matters because etcd backups are exactly the kind of artifact that drifts — copied to a debugging laptop, uploaded to a less-locked-down S3 bucket, kept around in a CI artifact store longer than intended. If that backup isn't encrypted, every Secret in the cluster is exposed the moment someone gets a copy of it.

Option A: Encryption at rest with EncryptionConfiguration

Kubernetes has a built-in mechanism for encrypting data in etcd: the EncryptionConfiguration resource, read by the API server at startup via the --encryption-provider-config flag. It tells the API server to encrypt specific resource types before writing them to etcd, and decrypt them transparently when read back through the API.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

A few things matter here: the order of providers determines both which one encrypts new writes (the first one) and which ones can still decrypt existing data (any listed provider). identity: {} means "no encryption" — keeping it listed after your real provider lets the API server still read any Secrets that were written before encryption was turned on, during migration. Generate the key with something like:

head -c 32 /dev/urandom | base64

Mount this config file and point the API server at it. On a kubeadm-managed control plane, that means editing the static pod manifest:

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
        # ...other flags
      volumeMounts:
        - name: encryption-config
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: encryption-config
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

On managed Kubernetes (EKS, GKE, AKS), you don't edit the API server directly — you don't have access to it. Instead, each cloud offers its own envelope encryption integration:

• EKS supports enabling "secrets encryption" at cluster creation or after, backed by a KMS key you specify — this configures aws-encryption-provider (a KMS plugin) behind the scenes, achieving the same envelope-encryption effect without you writing the EncryptionConfiguration file by hand.
• GKE offers "application-layer secrets encryption," which you enable per-cluster with a Cloud KMS key, documented via gcloud container clusters update --database-encryption-key=....
• AKS offers "etcd encryption" / customer-managed keys via Azure Key Vault integration, enabled at cluster creation.

In every case, the real pattern underneath is envelope encryption: Kubernetes encrypts each Secret with a local data-encryption key (DEK), and that DEK is itself encrypted by a key-encryption key (KEK) held in KMS/Key Vault — so the KMS key never directly touches your data, and you can rotate or revoke the KEK without re-encrypting every Secret from scratch.

What envelope encryption with a real KMS provider looks like

Rather than the static aescbc key above, production clusters typically use the kms provider in EncryptionConfiguration, which talks to an external KMS plugin over a Unix socket (KMS v2, the current API as of recent Kubernetes versions):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: myKmsPlugin
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}

The KMS plugin listening on that socket is what actually calls out to AWS KMS / GCP Cloud KMS / your HSM to encrypt and decrypt the DEK. This is exactly what the managed-cluster options above configure for you automatically — you're reading the same mechanism, just automated by the cloud provider's control plane.

Option B: External Secrets Operator — don't store the real secret in Kubernetes at all

Encryption at rest protects the Secret once it's in etcd. A different, complementary approach is to not put the real secret value in Kubernetes at all — keep it in a dedicated secret store (AWS Secrets Manager, GCP Secret Manager, Vault, Azure Key Vault) and have Kubernetes pull a synced copy only when needed. External Secrets Operator (ESO) is the standard open-source tool for this.

Install it with Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace

Define a SecretStore pointing at your backend — here, AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

That auth.jwt.serviceAccountRef pattern assumes IAM Roles for Service Accounts (IRSA) on EKS, so the operator authenticates to AWS without a static credential living in the cluster at all — worth setting up regardless of ESO, since it avoids long-lived AWS keys.

Then define an ExternalSecret that tells ESO which real secret to sync and what Kubernetes Secret to create from it:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/password
    - secretKey: username
      remoteRef:
        key: prod/db/username

ESO reconciles this on the refreshInterval schedule, calling out to Secrets Manager and writing (or updating) a normal db-credentials Kubernetes Secret that your pods consume exactly as before — no application code changes needed. The real secret's lifecycle (creation, rotation, deletion) lives entirely in Secrets Manager, where you likely already have better auditing and automated rotation than you'd build yourself in Kubernetes.

This doesn't replace encryption at rest — the synced copy is still a Secret object in etcd, still base64-only unless you've also enabled EncryptionConfiguration. ESO solves a different problem: keeping Kubernetes from being the system of record for the secret, so a leaked etcd snapshot only exposes a synced copy you can rotate immediately, not an originally-generated credential with no separate source to rotate it from.

Verify encryption is actually working

After enabling EncryptionConfiguration, confirm existing Secrets are actually re-encrypted — enabling it only affects new writes, not Secrets already sitting in etcd unencrypted. Force a rewrite of every existing Secret:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Then confirm directly against etcd (requires etcd access, typically from a control-plane node with the right certs):

ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-credentials \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C | head

If encryption is active, the value starts with a recognizable prefix like k8s:enc:kms:v2: or k8s:enc:aescbc:v1: followed by ciphertext — not readable JSON with a base64 password field sitting in it.

Common pitfalls

• Assuming base64 is encryption. It's reversible with no key, by design — it exists so binary-safe values fit in JSON/YAML, not for confidentiality.
• Enabling EncryptionConfiguration but never re-encrypting existing data. Old Secrets stay in their original plaintext (base64) form in etcd until something rewrites them.
• Putting identity: {} first in the providers list. That makes identity (no encryption) the write provider — new Secrets would be written unencrypted. The real encryption provider must be listed first.
• Treating ESO as a substitute for encryption at rest. ESO changes where the secret's source of truth lives; the synced Kubernetes Secret is still only as protected as your etcd encryption setup.
• Granting broad get/list on secrets via RBAC to service accounts or users who only need one specific Secret. Scope RBAC per-Secret with resourceNames where possible instead of blanket access to the whole resource type.

Troubleshooting

• API server fails to start after adding --encryption-provider-config: check the kubelet/API server logs for a malformed YAML error first — a bad indent in the EncryptionConfiguration file is the most common cause, and the key must be exactly 32 bytes before base64 encoding for aescbc.
• Secrets still show up unencrypted in etcd after enabling config: you forgot the rewrite step above — encryption only applies going forward from the config change.
• ESO's ExternalSecret never produces a Secret: run kubectl describe externalsecret db-credentials and check the status.conditions — the most common cause is a SecretStore auth failure, visible there with a clear error from the backend provider.
• ESO works but rotated values aren't showing up in pods: ESO updates the Secret object on its refresh interval, but your pod's environment variables were injected at container start — you still need a restart mechanism (see our Kopf operator post) or a sidecar that reloads config, exactly like the ConfigMap staleness problem.

Where to go next

If you're on a managed cluster, turn on the provider's built-in envelope encryption option first — it's the lowest-effort, highest-value fix, and it's usually one API call or console toggle. Then evaluate External Secrets Operator for anything where the secret's real source of truth should live outside Kubernetes, and pair both with tight RBAC on the secrets resource so encryption and operator access controls work together instead of either one being your only line of defense.

Mumuni Mohammed

Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.