A Kubernetes manifest is one of the easiest places to accidentally commit a secret. Someone pastes a real database password into a Secret object "just for local testing," means to remove it before committing, and doesn't. Once it's in git history, rotating the credential is the only real fix — deleting the file later doesn't delete the history. A pre-commit hook that catches this before the commit even lands is cheap insurance that costs you a few seconds on every commit.
This post builds a real one: a Python pre-commit hook that combines regex pattern matching for known secret shapes with Shannon entropy scoring to catch high-randomness strings that don't match any known pattern, specifically tuned to look inside Kubernetes manifest YAML.
1. Why regex alone isn't enough
Regex is great for secrets with a recognizable shape: an AWS access key always starts with AKIA followed by 16 characters, a GitHub token starts with ghp_, a Slack token starts with xox. These are easy, high-confidence catches.
But plenty of real secrets have no recognizable prefix — a raw database password, an internally generated API key, a base64-encoded blob in a Kubernetes Secret's data: field. Regex can't distinguish password123 (weak, not really a secret) from kj3$mP9!qT2vX8nL (looks exactly like a generated credential) by shape alone. For that, you need a measure of how random a string looks: Shannon entropy.
2. Shannon entropy, in plain terms
Shannon entropy measures how unpredictable a sequence of characters is, in bits per character. A string like aaaaaaaa has entropy near zero — every character is the same, so the next one is fully predictable. A string like kj3$mP9!qT2vX8nL has high entropy, close to the theoretical maximum for its character set, because each character carries close to full information — there's no pattern to exploit to predict the next one.
Real secrets — API keys, passwords, tokens generated by a CSPRNG — have high entropy by construction. English words, boilerplate YAML keys, and placeholder values have low-to-moderate entropy. Setting an entropy threshold lets you flag "this specific string looks like it was randomly generated" without needing to know its exact format in advance.
The formula for Shannon entropy of a string, given n occurrences of each distinct character out of N total characters:
import math
from collections import Counter
def shannon_entropy(data: str) -> float:
if not data:
return 0.0
counts = Counter(data)
length = len(data)
entropy = 0.0
for count in counts.values():
p = count / length
entropy -= p * math.log2(p)
return entropyA base64 alphabet (64 possible characters) has a theoretical max entropy of log2(64) = 6 bits/char; a hex string (16 possible characters) maxes at 4 bits/char. In practice, real secrets of 20+ characters tend to land above roughly 4.0-4.5 bits/char for base64-like strings, and that's a reasonable starting threshold to tune from.
3. The full hook: regex patterns plus entropy scanning
Here's a complete, runnable hook. It walks staged files, restricts itself to YAML (since that's what Kubernetes manifests are), and flags two kinds of findings: known-pattern matches and high-entropy strings found inside YAML string values.
#!/usr/bin/env python3
"""Pre-commit hook: detect likely secrets in Kubernetes manifests."""
import math
import re
import sys
from collections import Counter
ENTROPY_THRESHOLD = 4.3
MIN_SECRET_LENGTH = 20
KNOWN_PATTERNS = {
"AWS Access Key": re.compile(r"AKIA[0-9A-Z]{16}"),
"GitHub Token": re.compile(r"gh[pousr]_[A-Za-z0-9]{36,255}"),
"Slack Token": re.compile(r"xox[baprs]-[0-9A-Za-z-]{10,48}"),
"Generic Private Key": re.compile(r"-----BEGIN (RSA|EC|OPENSSH|PGP) PRIVATE KEY-----"),
"Google API Key": re.compile(r"AIza[0-9A-Za-z\-_]{35}"),
}
# Fields in Kubernetes manifests that commonly hold secret values directly
SUSPECT_KEY_PATTERN = re.compile(
r"^\s*(password|passwd|secret|token|apikey|api_key|access_key|"
r"client_secret|private_key)\s*:\s*[\"']?([^\s\"'#]+)",
re.IGNORECASE,
)
ALLOWLIST_PATTERN = re.compile(
r"\{\{.*\}\}|\$\(.*\)|<[A-Z_]+>|CHANGEME|REPLACE_ME|example\.com"
)
def shannon_entropy(data: str) -> float:
if not data:
return 0.0
counts = Counter(data)
length = len(data)
return -sum((c / length) * math.log2(c / length) for c in counts.values())
def scan_line(line: str, line_no: int, filename: str) -> list[str]:
findings = []
for name, pattern in KNOWN_PATTERNS.items():
if pattern.search(line):
findings.append(f"{filename}:{line_no}: possible {name} (pattern match)")
match = SUSPECT_KEY_PATTERN.match(line)
if match:
value = match.group(2)
if ALLOWLIST_PATTERN.search(value):
return findings
if len(value) >= MIN_SECRET_LENGTH:
score = shannon_entropy(value)
if score >= ENTROPY_THRESHOLD:
findings.append(
f"{filename}:{line_no}: high-entropy value on suspect key "
f"'{match.group(1)}' (entropy={score:.2f})"
)
return findings
def scan_file(path: str) -> list[str]:
findings = []
try:
with open(path, "r", encoding="utf-8", errors="ignore") as fh:
for line_no, line in enumerate(fh, start=1):
findings.extend(scan_line(line, line_no, path))
except (FileNotFoundError, IsADirectoryError):
pass
return findings
def main(argv: list[str]) -> int:
yaml_files = [f for f in argv if f.endswith((".yaml", ".yml"))]
all_findings = []
for path in yaml_files:
all_findings.extend(scan_file(path))
if all_findings:
print("Potential secrets found in staged Kubernetes manifests:\n")
for finding in all_findings:
print(f" {finding}")
print("\nRemove the secret, use a Kubernetes Secret reference, "
"External Secrets Operator, or Sealed Secrets instead, then re-commit.")
return 1
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))4. Why the hook targets data: and specific key names
A Kubernetes Secret manifest typically looks like this when someone has (incorrectly) hardcoded a real value instead of using a reference:
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
type: Opaque
stringData:
password: kj3$mP9!qT2vX8nL7wQ2
db_connection_token: AKIAIOSFODNN7EXAMPLEThe hook's SUSPECT_KEY_PATTERN specifically targets the YAML keys most likely to hold a secret value — password, token, secret, client_secret, and similar — rather than scanning every string in the file for entropy. Scanning blindly produces too many false positives: resource hashes, image digests, and generated names are also high-entropy, and they're not secrets. Anchoring the entropy check to a suspect key name keeps the signal-to-noise ratio workable.
5. Wiring it into pre-commit
Save the script as scripts/detect_secrets.py in your repo, then add it to your .pre-commit-config.yaml:
repos:
- repo: local
hooks:
- id: k8s-secret-scan
name: Scan Kubernetes manifests for secrets
entry: python3 scripts/detect_secrets.py
language: system
files: \.(yaml|yml)$
pass_filenames: trueInstall the pre-commit framework and the git hook itself once per clone:
pip install pre-commit
pre-commit installNow every git commit runs the scan against staged YAML files automatically, and a match blocks the commit with exit code 1 before it's created.
6. Handling legitimate false positives
You'll hit real false positives — a templated Helm value like {{ .Values.dbPassword }}, or a placeholder like CHANGEME. The ALLOWLIST_PATTERN in the script above skips common templating syntax and obvious placeholders. For anything else, add an inline allowlist comment convention and check for it before flagging:
if "# pragma: allowlist secret" in line:
return findingsUse this sparingly and require a PR reviewer to actually look at any line carrying that comment — it's an escape hatch, not a way to silence the tool permanently.
Common pitfalls
- Setting the entropy threshold too low. Below about 3.5 bits/char you'll flag ordinary English phrases and common config values constantly, and people will start ignoring the tool.
- Scanning every string in the file instead of anchoring to key names. This explodes false positives on hashes, UIDs, and image digests that are high-entropy but not secret.
- Only checking
git diffadditions, not full file content. If you only scan added lines, a secret added in one commit and never touched again in a later commit that re-triggers the hook can slip through on a rebase or squash. Scanning full staged file content avoids this. - Forgetting base64-encoded
data:fields. KubernetesSecretobjects under thedata:key (notstringData:) store base64-encoded values. Decode candidate values before entropy-scoring them, or you'll both miss real secrets and flag ordinary base64-encoded non-secrets inconsistently. - Treating this as sufficient on its own. A pre-commit hook only runs on the committer's machine and can be bypassed with
git commit --no-verify. Pair it with a server-side or CI-side scan (gitleaks, trufflehog, or your CI pipeline running the same script) as the real enforcement point.
Troubleshooting
- Hook doesn't run at all: confirm
pre-commit installwas run in this specific clone — it writes a local git hook file and doesn't carry over to a fresh clone automatically. - Hook runs but never finds anything, even on a test secret: check that your test value is actually ≥
MIN_SECRET_LENGTHcharacters and uses the full key range (mixed case, digits, symbols) — a short or low-variety string won't cross the entropy threshold. - Too many false positives on CI-generated manifests: add the specific key patterns or filenames producing noise to an exclusion list, rather than lowering the global entropy threshold.
Where to go next
Once this is in place, look at purpose-built tools like gitleaks or detect-secrets for broader pattern coverage and baseline management, and pair the pre-commit hook with a server-side check in CI so the control doesn't depend on every contributor having the hook installed locally.
Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.
