Turning on a secret scanner for the first time against an existing repository almost always produces one result: hundreds of findings, most of them years old, most of them not worth anyone's time to re-triage from scratch. That's the problem detect-secrets (originally built at Yelp, pure Python) solves differently from a plain regex scanner — it separates "here's everything that looks like a secret" from "here's what's new since we last checked," using a committed baseline file as the dividing line. This tutorial sets up that baseline workflow, wires it into pre-commit, and then goes further: writing a custom detector plugin in Python for a secret format the built-in rules don't know about.
Step 1: Install and create a baseline
pip install detect-secrets
detect-secrets scan > .secrets.baseline
This scans the repository once and records every match as a baseline — not as "approved," just as "known, as of now." Commit .secrets.baseline to the repo. From this point on, the question a scan answers isn't "are there secrets in this repo" (there might legitimately be some, in test fixtures, forever) — it's "is there anything new."
Step 2: Triage the baseline
The baseline file lists findings but doesn't say whether each one is a real secret or a false positive yet. Audit it interactively:
detect-secrets audit .secrets.baseline
This walks through each finding one at a time, showing the surrounding code, and asks you to mark it real or a false positive. The result is saved back into the baseline file itself — each entry gets an is_secret field once triaged. This is a one-time cost per existing finding; it's the step that turns "hundreds of alerts" into "a reviewed, trusted baseline."
Step 3: Catch new secrets in pre-commit
# .pre-commit-config.yaml
repos:
- repo: https://github.com/Yelp/detect-secrets
rev: v1.5.0
hooks:
- id: detect-secrets
args: ['--baseline', '.secrets.baseline']
exclude: package-lock.json
With the baseline in place, this hook only fails a commit for a finding that isn't already in the baseline — existing, triaged findings don't block anyone. A genuinely new secret does. Run pre-commit install once per clone to activate it.
Step 4: Keep the baseline current
Code changes, and the baseline needs to track it — re-run the scan periodically (or in CI) to pick up new findings that need triage:
detect-secrets scan --baseline .secrets.baseline
This updates the existing baseline in place rather than replacing it — already-triaged entries keep their is_secret status, and only genuinely new matches get added as untriaged.
Step 5: Write a custom plugin for your own secret format
The built-in plugins cover well-known formats (AWS keys, high-entropy strings, private keys, Slack tokens, and more), but a growing codebase accumulates its own internal token formats that no generic rule will catch. detect-secrets' plugin base class makes adding one straightforward — subclass RegexBasedDetector and define a secret_type name and a denylist of compiled patterns:
# internal_token_plugin.py
import re
from detect_secrets.plugins.base import RegexBasedDetector
class InternalServiceTokenDetector(RegexBasedDetector):
"""Flags our internal service tokens: isvc_live_<32 hex chars>."""
secret_type = "Internal Service Token"
denylist = [
re.compile(r'isvc_live_[a-f0-9]{32}'),
re.compile(r'isvc_test_[a-f0-9]{32}'),
]
This follows the same pattern as detect-secrets' own built-in plugins (several of them, like the AWS key detector, are themselves thin RegexBasedDetector subclasses) — a class that scans each line against one or more compiled regexes and reports a match as a potential secret.
To run it without forking the project, the supported path is registering it as a plugin detect-secrets can discover — via a setuptools entry point in your own package's pyproject.toml, under the plugin discovery group detect-secrets exposes for third-party detectors. The exact entry-point group name has shifted slightly across detect-secrets versions, so check it against the plugins module of the version you've pinned before wiring this into CI — but the shape of it is: your package declares the entry point, points it at InternalServiceTokenDetector, and once installed alongside detect-secrets, detect-secrets scan picks it up automatically next to the built-in plugins, no source changes to detect-secrets itself required.
For a quick local test of the detector logic alone, without worrying about plugin registration yet, you can exercise the class directly:
detector = InternalServiceTokenDetector()
line = 'TOKEN = "isvc_live_4f9a2c8e1b3d7f0a6c5e9b2d8f1a4c7e"'
secrets = detector.analyze_line(filename="config.py", line=line, line_number=1)
for secret in secrets:
print(secret.secret_type, secret.secret_hash)
This is useful on its own as a fast feedback loop while writing the regex — confirm it matches what it should and (just as important) doesn't match things it shouldn't, before wiring it into the full scan pipeline.
Step 6: Run it in CI with a hard fail on new findings
name: secret-scan
on:
pull_request:
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install detect-secrets
- name: Check for new secrets
run: |
detect-secrets scan --baseline .secrets.baseline
git diff --exit-code .secrets.baseline
The trick here is the last line, not the scan itself: re-running the scan against the committed baseline and then checking whether the baseline file changed. If the scan found something not already in the baseline, the baseline file gets a new untriaged entry, git diff sees that change, and --exit-code turns it into a CI failure — a genuinely new secret blocks the PR, without needing a separate mechanism to distinguish "new" from "already known."
Common pitfalls
- Never triaging the initial baseline. Every entry sits as
is_secret: nullforever, and the baseline stops meaning anything — the whole value of this workflow is in the one-time audit. - Forgetting to commit baseline updates. If a developer runs
detect-secrets scan --baseline ...locally and doesn't commit the result, the next person's scan looks like it has new findings that are actually already resolved on disk but not in version control. - A custom plugin regex that's too loose. Test it against real examples of what should and shouldn't match before registering it — a greedy pattern turns every scan into a wall of false positives, which trains people to ignore the tool entirely.
- Excluding lockfiles incorrectly.
package-lock.json/poetry.lock/similar files can contain high-entropy strings that aren't secrets (hashes, resolved URLs) — exclude them explicitly rather than letting them generate noise in every scan.
Where to go next
Once the baseline workflow is solid, the same `detect-secrets audit` command supports a --report mode that summarizes baseline health across a repo — useful as a periodic check that the baseline isn't quietly accumulating untriaged entries between audits.
Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.
