Helm Charts 101 — What Actually Happens When You helm install

I've debugged more "why didn't my config change take effect" tickets caused by Helm's release bookkeeping than by Kubernetes itself. Here's what actually happens under the hood, with a real chart you can build alongside me.

Most Helm tutorials stop at helm create mychart and helm install. That's fine for getting something running, but it leaves a gap: the moment something goes wrong — a value that "didn't apply," a rollback that didn't roll back, a CRD that vanished on upgrade — you're stuck, because you never saw what Helm actually does between the command and the cluster.

This post walks through that gap. We'll build a small chart from scratch, render it, install it, inspect exactly what Helm stored in the cluster to remember what it did, and then break a rollback on purpose so you can see why it behaves the way it does.

What a chart actually is on disk

A Helm chart is not a Kubernetes-native concept. Kubernetes has no idea Helm exists — Helm is a client-side templating and bookkeeping tool that happens to talk to the same API server as kubectl. A chart is just a directory with a fixed layout:

mychart/
  Chart.yaml          # name, version, apiVersion — metadata only
  values.yaml          # default values, overridable at install/upgrade time
  charts/               # vendored sub-charts (dependencies), each a full chart itself
  templates/            # Go templates that render into Kubernetes YAML
    deployment.yaml
    service.yaml
    _helpers.tpl        # named templates, no leading/trailing --- of their own
    NOTES.txt            # printed after install/upgrade, also templated
  crds/                 # CustomResourceDefinitions, handled specially (more below)

Let's build one. This is a small app chart with a Deployment, a Service, and a post-install smoke-test Job — enough surface area to see the interesting parts.

# Chart.yaml
apiVersion: v2
name: mychart
description: A minimal chart for walking through what helm install does
type: application
version: 0.1.0
appVersion: "1.0.0"

# values.yaml
replicaCount: 2

image:
  repository: nginx
  tag: "1.27"

service:
  port: 80

resources:
  limits:
    cpu: 200m
    memory: 128Mi

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
  labels:
    app: {{ .Chart.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  template:
    metadata:
      labels:
        app: {{ .Chart.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: 80
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

# templates/_helpers.tpl
{{- define "mychart.fullname" -}}
{{ .Release.Name }}-{{ .Chart.Name }}
{{- end -}}

Two things worth noticing already. First, {{ include "mychart.fullname" . }} is not string interpolation of a value — it's calling a named template defined in _helpers.tpl, which itself references .Release.Name. .Release is not in values.yaml at all; it's part of the built-in context Helm injects at render time, alongside .Chart (parsed from Chart.yaml) and .Capabilities (what the target cluster supports). Second, toYaml .Values.resources | nindent 12 is doing real work: it serializes a nested map back into YAML text and re-indents it, because Go templates have no concept of YAML structure — they're producing text, and it's on you to keep that text valid YAML.

Rendering is a local, offline step — mostly

Before anything touches the cluster, Helm renders every file under templates/ as a Go template, using Sprig's extra functions (toYaml, nindent, default, required, and friends) plus three built-in objects: .Values, .Chart, .Release, and .Capabilities. You can run exactly this step with no cluster involved at all:

helm template myrelease ./mychart --set replicaCount=3

# output (abridged)
---
# Source: mychart/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myrelease-mychart
  labels:
    app: mychart
spec:
  replicas: 3
  ...

Note the # Source: comment — Helm concatenates every rendered template into one YAML stream separated by ---, and tags each block with which file it came from. That's the exact text Helm is about to hand to the API server; if a value "didn't apply," helm template with the same flags you used for install is the first thing to run, because it shows you definitively whether the problem is in rendering or somewhere else.

Values merge in a specific priority order, highest wins:

1. --set / --set-string / --set-json flags (highest)
2. -f / --values files, in the order given on the command line (later files win)
3. the chart's own values.yaml (lowest — the fallback default)

Sub-charts add one more wrinkle: a parent chart's values.yaml can set values under a key matching the sub-chart's name, and those get merged into that sub-chart's own defaults. It's a common source of "I set it in the parent and it didn't take" bugs when the key name doesn't exactly match the sub-chart's directory/name.

One nuance: helm install --dry-run is not purely local the way helm template is. By default it still contacts the cluster to fetch .Capabilities (API versions, Kubernetes version) and, depending on your Helm version, to validate the rendered manifests server-side without persisting anything. If you want a guaranteed zero-cluster-contact render, use helm template, not --dry-run.

What helm install actually does, step by step

Once rendering produces a final manifest, here's the sequence that actually happens:

1. Render every template in templates/ into one combined YAML document, as above.
2. Validate the result against the Kubernetes OpenAPI schema for the target cluster (this is why a typo'd field name fails before anything is created).
3. Apply CRDs from crds/ first, if any exist, using plain kubectl apply-equivalent semantics — not templated, no values substitution, installed only, never upgraded or deleted by Helm after the first install. This trips people up constantly: if you change a CRD's schema and bump the chart version, helm upgrade will not touch the CRD. You have to apply it yourself.
4. Create the Kubernetes objects from the rendered manifest, in dependency order (namespaces and CRDs before objects that need them).
5. Run hooks at their designated points in the lifecycle (see below) — these are ordinary Kubernetes Jobs/Pods, not part of the same atomic transaction as step 4.
6. Record the release. This is the part most people never look at, and it's the key to everything Helm does afterward.

The release record: Helm's actual memory

Helm 3 has no server-side component (no Tiller, unlike Helm 2). So where does it remember "release myrelease, revision 1, is currently these exact manifests with these exact values"? It stores that as a Kubernetes Secret, right in your cluster, in the same namespace as the release.

helm install myrelease ./mychart --set replicaCount=3
kubectl get secrets -l owner=helm,name=myrelease

NAME                           TYPE                 DATA   AGE
sh.helm.release.v1.myrelease.v1   helm.sh/release.v1   1      5s

That Secret's single data field contains the entire release object — the rendered manifest, the values used, the chart metadata — serialized to JSON, gzip-compressed, and base64-encoded (then base64-encoded again by Kubernetes itself, since Secret data is always base64 in the API). You can decode it yourself:

kubectl get secret sh.helm.release.v1.myrelease.v1 -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gzip -d | python3 -m json.tool | head -40

This is also exactly what helm get manifest myrelease, helm get values myrelease, and helm history myrelease are reading — they're not asking the cluster "what's actually running," they're reading Helm's own stored record of what it thinks it last applied. That distinction matters: if someone kubectl edits a Deployment that Helm manages, Helm's record doesn't know. The drift is invisible to helm get manifest until the next helm upgrade, which is where the three-way merge comes in.

Every revision gets its own Secret (...v1, ...v2, ...), labeled with status: deployed | superseded | failed | pending-upgrade. By default Helm keeps the last 10 revisions per release and prunes older ones — pass --history-max to install/upgrade to change that. This is also why helm rollback myrelease 1 can fail with "release not found" for a revision — if it's been pruned, the record is simply gone.

Upgrades use a three-way merge, not a blind re-apply

This is the part that causes the most confusing bugs. When you run helm upgrade, Helm computes a patch from three inputs, not two:

• the previous rendered manifest (from the stored release Secret)
• the new rendered manifest (from this upgrade)
• the live state currently in the cluster

It diffs old→new to figure out your intent, then applies that as a strategic merge patch against live state. This is why manually editing a Helm-managed resource with kubectl edit or kubectl scale can get silently reverted on the next helm upgrade — Helm sees that the field existed in the old chart render, isn't in your manual edit's provenance, and reconciles it back toward what the chart says it should be. If you need a field Helm doesn't manage, templating it as an empty/omitted value in the chart doesn't protect it; you either need the chart to actually own that field or manage it outside Helm entirely.

Hooks run outside the atomic apply

Add a hook to see this directly. Here's a post-install smoke test as a Job:

# templates/tests/smoke-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "mychart.fullname" . }}-smoke-test
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke-test
          image: curlimages/curl:8.10.1
          command:
            - sh
            - -c
            - "curl -sf http://{{ include \"mychart.fullname\" . }}:{{ .Values.service.port }}"

Three annotations are doing the real work here:

• helm.sh/hook — which lifecycle points trigger it. post-install and post-upgrade run after the main manifest is applied and considered ready, not as part of it.
• helm.sh/hook-weight — hooks at the same point run in ascending weight order (ties broken by name), so you can sequence multiple hooks (e.g. a schema-migration Job at weight -5 before a smoke test at weight 0).
• helm.sh/hook-delete-policy — without this, completed hook Jobs just pile up in the cluster forever. before-hook-creation deletes the previous hook resource right before creating a new one (so re-running install/upgrade doesn't collide on the same Job name); hook-succeeded deletes it immediately after it succeeds.

The consequence people miss: hooks are not transactional with the rest of the release. If your post-install hook fails, the Deployment and Service from step 4 are still sitting there, already created — Helm marks the release failed, but it does not automatically tear down what it already applied. --atomic on helm install/upgrade changes this: on any failure (hook or otherwise) it triggers an automatic rollback to the last known-good revision, which is the closest Helm gets to "all or nothing."

Rollback: replaying a stored manifest, not undoing actions

Try it: break the chart (set the image tag to something that doesn't exist), upgrade, watch it fail, then roll back.

helm upgrade myrelease ./mychart --set image.tag=does-not-exist
helm rollback myrelease 1

helm rollback does not "undo" anything in a generic sense — it takes the manifest stored in the target revision's release Secret and applies that manifest as a new revision (so a rollback to revision 1 becomes revision 3, not a reversion to revision 1 itself), using the same three-way merge logic as a normal upgrade. One sharp edge: hooks registered as pre-rollback/post-rollback are a distinct set from pre-upgrade/post-upgrade — if your migration logic only lives in a pre-upgrade hook, rolling back will not run it in reverse, because nothing runs it at all unless you also defined a pre-rollback/post-rollback hook with that logic.

A quick checklist for when something's "not applying"

• helm template with your exact flags — confirms rendering produced what you expect, with zero cluster involvement.
• helm get values myrelease — shows the merged values Helm actually used for the current revision, so you can see if a flag or file didn't take effect in the merge order.
• helm get manifest myrelease — the manifest Helm believes it applied; compare against kubectl get <resource> -o yaml to catch drift.
• helm history myrelease — every revision with its status; useful for finding exactly when a failed or superseded revision happened.
• If a CRD change isn't showing up: remember crds/ is install-only. Apply the updated CRD yourself.

None of this requires memorizing internals — it requires knowing that Helm's state lives in a Secret you can read, that rendering and applying are separate steps you can run independently, and that upgrades are a three-way merge, not a blind replace. Once that clicks, most "Helm is being weird" tickets turn into a two-minute kubectl get secret away from an answer.

Mumuni Mohammed

Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.