If a ConfigMap changes and the pods reading it don't restart, those pods keep running on stale config until something happens to bounce them. Kubernetes won't do that restart for you. This post builds a small, real Kubernetes operator in Python with Kopf that watches a ConfigMap and restarts the Deployment pods that depend on it.
What an operator actually does, and why Kopf
A Kubernetes operator is a controller that watches the state of objects in the cluster (through the API server's watch mechanism) and takes action to reconcile reality with some desired state. Most production operators are written in Go using controller-runtime, but Kopf (Kubernetes Operator Pythonic Framework) lets you write the same reconciliation pattern in plain Python, using decorators to register handlers for events on specific resource kinds. It's a solid choice when your team is Python-first, the logic is more about orchestration/business rules than raw throughput, and you don't need a custom CRD with generated clients.
Kopf handles the hard parts for you: setting up watches with resync and retry logic, handling reconnects, tracking per-object state across restarts (via annotations it manages), and giving you clean async-friendly handler functions.
The problem we're solving
Say you have a Deployment whose pods mount a ConfigMap as environment variables or a volume. Kubernetes propagates volume-mounted ConfigMap changes to the container's filesystem eventually (kubelet syncs it, typically within a minute, cache permitting), but:
- If the ConfigMap is consumed as environment variables (
envFromorvalueFrom.configMapKeyRef), the values are injected at container start and never update until the pod restarts, full stop. - Even for volume mounts, your application process usually doesn't know to reload the file — it read it once at startup.
The operator's job: watch for a labeled ConfigMap to change, find the Deployments that declare a dependency on it (via an annotation), and trigger a rolling restart.
Step 1 — Install Kopf and set up the project
python3 -m venv venv
source venv/bin/activate
pip install kopf kubernetes
Kopf needs RBAC permissions to watch ConfigMaps and patch Deployments. If you're running it inside the cluster (recommended for anything beyond local testing), you'll need a ServiceAccount, ClusterRole, and ClusterRoleBinding — shown in Step 4.
Step 2 — Define the dependency convention
We need a way for a Deployment to declare "I depend on this ConfigMap." The simplest, most explicit approach: an annotation on the Deployment naming the ConfigMap it depends on.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: default
annotations:
configwatch.ktechhub.io/configmap: app-config
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web-app
image: myregistry/web-app:1.4.0
envFrom:
- configMapRef:
name: app-config
Any Deployment carrying configwatch.ktechhub.io/configmap: <name> is one our operator should restart when that ConfigMap changes, within the same namespace.
Step 3 — Write the operator
Kopf handlers are registered with decorators like @kopf.on.update, @kopf.on.create, and @kopf.on.field, scoped to a specific API group/version/resource. For a built-in resource like ConfigMap, the group is empty and version is v1.
import kopf
import kubernetes
import hashlib
import json
import logging
logger = logging.getLogger(__name__)
DEPENDENCY_ANNOTATION = "configwatch.ktechhub.io/configmap"
RESTART_ANNOTATION = "configwatch.ktechhub.io/restartedAt"
def _config_hash(data: dict) -> str:
"""Stable hash of ConfigMap data, used to detect real content changes."""
serialized = json.dumps(data or {}, sort_keys=True)
return hashlib.sha256(serialized.encode()).hexdigest()[:16]
@kopf.on.update("", "v1", "configmaps")
def on_configmap_update(spec, old, new, name, namespace, logger, **kwargs):
"""Fires whenever a ConfigMap object changes in any watched namespace."""
old_data = (old or {}).get("data", {})
new_data = (new or {}).get("data", {})
old_hash = _config_hash(old_data)
new_hash = _config_hash(new_data)
if old_hash == new_hash:
logger.debug("ConfigMap %s/%s changed but data hash is unchanged, skipping", namespace, name)
return
logger.info("ConfigMap %s/%s data changed (%s -> %s), looking for dependents",
namespace, name, old_hash, new_hash)
apps_v1 = kubernetes.client.AppsV1Api()
deployments = apps_v1.list_namespaced_deployment(namespace=namespace)
restarted = []
for dep in deployments.items:
annotations = dep.metadata.annotations or {}
if annotations.get(DEPENDENCY_ANNOTATION) == name:
_restart_deployment(apps_v1, dep.metadata.name, namespace, logger)
restarted.append(dep.metadata.name)
if not restarted:
logger.info("No Deployments in %s depend on ConfigMap %s", namespace, name)
return {"restarted_deployments": restarted}
def _restart_deployment(apps_v1, deployment_name, namespace, logger):
"""Triggers a rolling restart the same way `kubectl rollout restart` does:
patch a pod-template annotation so the Deployment's pod spec hash changes."""
import datetime
timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
patch_body = {
"spec": {
"template": {
"metadata": {
"annotations": {
RESTART_ANNOTATION: timestamp
}
}
}
}
}
apps_v1.patch_namespaced_deployment(
name=deployment_name,
namespace=namespace,
body=patch_body,
)
logger.info("Restarted Deployment %s/%s due to ConfigMap dependency change",
namespace, deployment_name)
@kopf.on.startup()
def configure(settings: kopf.OperatorSettings, **kwargs):
# Keep Kopf's own annotations/finalizers tidy and avoid noisy diff-base storage.
settings.persistence.finalizer = "configwatch.ktechhub.io/finalizer"
settings.posting.level = logging.INFO
The key trick is in _restart_deployment: Kubernetes doesn't have a direct "restart this Deployment" API call. kubectl rollout restart itself works by patching an annotation on the pod template, which changes the template's content, which the Deployment controller detects as a change requiring a new ReplicaSet — exactly what we do here.
We hash the ConfigMap's data field and compare old vs. new rather than reacting to every on.update event, because Kubernetes fires update events for metadata-only changes too (like resourceVersion bumps from unrelated causes), and we don't want to restart pods for no-op changes.
Step 4 — RBAC for running inside the cluster
The operator needs to read ConfigMaps and patch Deployments cluster-wide (or scoped to specific namespaces, if you want to be tighter):
apiVersion: v1
kind: ServiceAccount
metadata:
name: configwatch-operator
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: configwatch-operator
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: ["kopf.dev"]
resources: ["clusterkopfpeerings"]
verbs: ["list", "watch", "patch", "get"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["list", "watch", "get"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: configwatch-operator
subjects:
- kind: ServiceAccount
name: configwatch-operator
namespace: kube-system
roleRef:
kind: ClusterRole
name: configwatch-operator
apiGroup: rbac.authorization.k8s.io
The clusterkopfpeerings and customresourcedefinitions permissions are standard Kopf plumbing — Kopf uses peering objects to coordinate when multiple operator replicas are running, and checks for CRDs related to its own operation. You can skip the peering CRD and permission entirely if you only ever run a single operator replica, which is fine for most teams starting out.
Step 5 — Package and deploy it
configwatch-operator/
├── operator.py
├── Dockerfile
└── requirements.txt
# requirements.txt
kopf==1.37.2
kubernetes==30.1.0
# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY operator.py .
CMD ["kopf", "run", "operator.py", "--all-namespaces"]
apiVersion: apps/v1
kind: Deployment
metadata:
name: configwatch-operator
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: configwatch-operator
template:
metadata:
labels:
app: configwatch-operator
spec:
serviceAccountName: configwatch-operator
containers:
- name: operator
image: myregistry/configwatch-operator:0.1.0
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 128Mi
kopf run operator.py --all-namespaces is the actual entrypoint Kopf expects — it's a CLI tool that imports your handler module and starts the watch loops. You could instead scope it to specific namespaces with --namespace if you don't want cluster-wide ConfigMap watches.
Testing it locally
You don't need to deploy to test logic. Point Kopf at any cluster your kubectl context can reach (a local kind/minikube cluster is ideal) and run it directly:
kopf run operator.py --verbose
Then in another terminal, create the dependent Deployment and ConfigMap from Step 2, and edit the ConfigMap:
kubectl create configmap app-config --from-literal=LOG_LEVEL=info
kubectl apply -f web-app-deployment.yaml
# trigger a change
kubectl patch configmap app-config --type merge -p '{"data":{"LOG_LEVEL":"debug"}}'
# watch the operator's terminal log the restart, then confirm:
kubectl get pods -l app=web-app -w
You should see new pods roll in within a few seconds of the patch, and the operator's log should show Restarted Deployment default/web-app due to ConfigMap dependency change.
Common pitfalls
- Reacting to every update event, not just data changes. Without the hash comparison, you'll restart deployments on metadata-only churn — label updates, annotation bumps from other controllers, even Kubernetes' own periodic resync. Always diff on the fields that actually matter.
- Forgetting
binaryData. ConfigMaps can hold binary content in a separatebinaryDatafield. If your ConfigMaps ever use it, hash that too, or you'll miss real changes. - Missing RBAC for
patchon deployments. A common first error is read access (get/list/watch) without write access, so the operator can detect the change but fails silently or loudly when it tries to restart. - Not handling the Kopf peering requirement correctly when running more than one replica. Two operator instances without peering configured can race and double-process events, or one can silently stop processing if peering is misconfigured. Stick to one replica unless you specifically need HA and have set up peering.
- Assuming this also handles Secrets. This operator only watches ConfigMaps. Secrets consumed as env vars have the same stale-value problem, and you'd extend the same pattern to
@kopf.on.update("", "v1", "secrets")if needed — but be careful about logging anything from a Secret's data.
Troubleshooting
- Operator doesn't see any events at all: confirm the ServiceAccount's ClusterRole actually grants
watchonconfigmaps, and check the pod's logs for a 403 Forbidden from the API server — that's the first thing to rule out. - Restarts aren't happening even though the ConfigMap clearly changed: add a debug log of both hashes right before the comparison, and confirm the Deployment's annotation value exactly matches the ConfigMap's
metadata.name(case-sensitive, no typos). - Operator crashes on startup with a Kopf peering error: either apply the
ClusterKopfPeeringCRD and object Kopf expects, or disable peering by settingsettings.peering.standalone = Truein the@kopf.on.startup()handler if you only run one replica. - Pods restart but still have the old values: check whether the Deployment is actually mounting the ConfigMap via
envFrom/valueFrom(as expected) versus reading it from a mounted volume cached at a layer your restart doesn't touch — in rare setups, an init container or sidecar caches config separately.
Where to go next
This pattern generalizes well beyond ConfigMaps: the same hash-and-patch approach works for Secrets, and the same "watch X, act on Y" shape is the basis for most real operators. From here, look at turning the dependency declaration into a proper Custom Resource Definition instead of an annotation convention — Kopf makes handling CRDs with @kopf.on.create/@kopf.on.update on your own API group just as easy, and it gives you schema validation for free via the CRD's OpenAPI schema.
Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.
