Building a Python Prometheus Exporter for Custom Security and Ops Metrics

Prometheus ships exporters for the obvious things — CPU, memory, HTTP request latency. It has no idea your TLS certificate expires in six days, or that someone just failed to log in forty times in a minute. Those are metrics only your application logic knows about, and if you want them in Prometheus, you have to expose them yourself.

This post builds a real Prometheus exporter inside a FastAPI app: a /metrics endpoint backed by the official prometheus_client library, exposing a certificate-expiry gauge and a failed-auth-attempts counter alongside the app's normal routes.

1. The Prometheus client model: Counter, Gauge, Histogram

The prometheus_client library gives you a handful of metric types, and picking the right one matters because Prometheus's query language (PromQL) expects specific semantics from each:

• Counter — a value that only goes up (or resets to zero on restart). Use it for things like total failed auth attempts, total requests served.
• Gauge — a value that goes up or down. Use it for things like certificate days-until-expiry, current active connections, current queue depth.
• Histogram — buckets observations into ranges, used for things like request latency distributions.

Install the library alongside FastAPI:

pip install fastapi uvicorn prometheus-client cryptography

2. Setting up the metrics and the /metrics endpoint

Prometheus scrapes a single plain-text endpoint on your service, conventionally /metrics, and expects a specific content type. prometheus_client handles the formatting; you just need to wire it into a FastAPI route:

from fastapi import FastAPI, Response
from prometheus_client import Counter, Gauge, generate_latest, CONTENT_TYPE_LATEST

app = FastAPI()

# --- Security/ops metrics ---
FAILED_AUTH_ATTEMPTS = Counter(
    "app_failed_auth_attempts_total",
    "Total number of failed authentication attempts",
    ["reason"],
)

CERT_EXPIRY_SECONDS = Gauge(
    "app_tls_cert_expiry_seconds",
    "Seconds until the TLS certificate expires",
    ["cert_name"],
)

ACTIVE_SESSIONS = Gauge(
    "app_active_sessions",
    "Current number of active authenticated sessions",
)


@app.get("/metrics")
def metrics():
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)

Note the metric naming convention: app_<noun>_<unit>, with a _total suffix for counters and the unit spelled out (_seconds, not an ambiguous number). This matches Prometheus's own naming guidelines and makes the metric self-documenting in Grafana later.

3. Wiring the failed-auth counter into real auth logic

The counter needs to be incremented exactly where a failed auth actually happens, with a label distinguishing why it failed — bad password versus expired token versus unknown user all point to different follow-up actions:

from fastapi import HTTPException, Depends
from pydantic import BaseModel

class LoginRequest(BaseModel):
    username: str
    password: str

def verify_credentials(username: str, password: str) -> bool:
    # Replace with your real credential check
    return username == "admin" and password == "correct-password"

@app.post("/login")
def login(req: LoginRequest):
    if not verify_credentials(req.username, req.password):
        FAILED_AUTH_ATTEMPTS.labels(reason="bad_credentials").inc()
        raise HTTPException(status_code=401, detail="Invalid credentials")
    ACTIVE_SESSIONS.inc()
    return {"status": "ok"}

Labels are powerful but dangerous: each unique combination of label values creates a new time series in Prometheus's storage. Never put a username, IP address, or anything high-cardinality into a label — reason with a handful of fixed values (bad_credentials, expired_token, unknown_user, rate_limited) is safe; a raw username is not, and can quietly blow up your Prometheus storage.

4. Certificate expiry as a gauge, refreshed on a schedule

Certificate expiry isn't something you compute per-request — it's an external fact you check periodically and publish as a gauge. Use the cryptography library to read a cert file's expiry, and a background task to refresh the gauge:

import asyncio
from datetime import datetime, timezone
from cryptography import x509
from cryptography.hazmat.backends import default_backend

CERT_PATHS = {
    "api-gateway": "/etc/ssl/certs/api-gateway.pem",
    "internal-mtls": "/etc/ssl/certs/internal-mtls.pem",
}

def read_cert_expiry_seconds(path: str) -> float:
    with open(path, "rb") as f:
        cert = x509.load_pem_x509_certificate(f.read(), default_backend())
    expires_at = cert.not_valid_after_utc
    now = datetime.now(timezone.utc)
    return (expires_at - now).total_seconds()

async def refresh_cert_metrics():
    while True:
        for name, path in CERT_PATHS.items():
            try:
                seconds_left = read_cert_expiry_seconds(path)
                CERT_EXPIRY_SECONDS.labels(cert_name=name).set(seconds_left)
            except (FileNotFoundError, ValueError):
                CERT_EXPIRY_SECONDS.labels(cert_name=name).set(-1)
        await asyncio.sleep(3600)  # refresh hourly

@app.on_event("startup")
async def start_background_tasks():
    asyncio.create_task(refresh_cert_metrics())

Setting the gauge to -1 on a read failure (missing file, unparseable cert) is deliberate: it's a value that's obviously not a real "seconds remaining," and you can alert on app_tls_cert_expiry_seconds < 0 or app_tls_cert_expiry_seconds < 604800 (under zero, or under 7 days) in the same rule.

Note: @app.on_event("startup") is deprecated in recent FastAPI versions in favor of a lifespan context manager; use whichever your FastAPI version documents, the metric logic itself is unaffected.

5. Running it and scraping it

uvicorn main:app --host 0.0.0.0 --port 8000

curl http://localhost:8000/metrics

You should see Prometheus's text exposition format:

# HELP app_failed_auth_attempts_total Total number of failed authentication attempts
# TYPE app_failed_auth_attempts_total counter
app_failed_auth_attempts_total{reason="bad_credentials"} 3.0
# HELP app_tls_cert_expiry_seconds Seconds until the TLS certificate expires
# TYPE app_tls_cert_expiry_seconds gauge
app_tls_cert_expiry_seconds{cert_name="api-gateway"} 2591382.0
app_tls_cert_expiry_seconds{cert_name="internal-mtls"} 518221.0
# HELP app_active_sessions Current number of active authenticated sessions
# TYPE app_active_sessions gauge
app_active_sessions 1.0

Point a Prometheus server at it with a scrape config:

scrape_configs:
  - job_name: "my-fastapi-app"
    scrape_interval: 30s
    static_configs:
      - targets: ["my-fastapi-app:8000"]

6. Alerting rules worth writing immediately

Two rules that justify the whole exercise on their own:

groups:
  - name: security-ops
    rules:
      - alert: CertExpiringSoon
        expr: app_tls_cert_expiry_seconds > 0 and app_tls_cert_expiry_seconds < 604800
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate {{ $labels.cert_name }} expires in under 7 days"

      - alert: HighFailedAuthRate
        expr: rate(app_failed_auth_attempts_total[5m]) > 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Failed auth rate on {{ $labels.reason }} exceeds 1/sec over 5m"

The rate() function is why Counter has to be monotonically increasing — Prometheus computes the per-second rate of increase over the window, which is meaningless if the raw value can decrease on its own between restarts.

7. Multi-process deployments: a real gotcha

If you run this app under Gunicorn with multiple worker processes, the default prometheus_client registry is per-process — each worker has its own in-memory counters, and a scrape hitting one worker won't see another worker's increments. For multi-process deployments, use the library's documented multiprocess mode, which writes metrics to files in a shared directory and aggregates them on scrape:

export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_multiproc

from prometheus_client import multiprocess, CollectorRegistry

@app.get("/metrics")
def metrics():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return Response(content=generate_latest(registry), media_type=CONTENT_TYPE_LATEST)

Skipping this step is the single most common reason a working-in-development exporter reports wrong numbers in production behind a multi-worker server.

Common pitfalls

• High-cardinality labels. Usernames, IPs, request IDs as label values will degrade your Prometheus instance over time as the number of distinct time series grows unbounded.
• Using a Gauge for something that should be a Counter, or vice versa. A Counter that can decrease (say, you used .set() on it instead of .inc()) breaks rate() calculations silently.
• Blocking the event loop in the metrics refresh task. Reading a cert file is cheap, but if you add anything slower (a network call, a DB query) inside an async def background task without awaiting properly, you can stall request handling.
• Forgetting multiprocess mode under Gunicorn/multiple workers. Metrics will look fine locally with uvicorn and wrong in production.
• Not securing /metrics. It's often left open on the same port as the app with no auth, and it can leak structural information about the app (label values, internal names). Restrict it at the network layer or put it behind an internal-only port.

Troubleshooting

• Prometheus shows the target as down: confirm the content type returned matches CONTENT_TYPE_LATEST exactly, and that the endpoint isn't behind auth middleware that Prometheus's scrape config isn't configured to pass.
• Counter values reset unexpectedly: this is normal on app restart — Counters are in-memory and reset to zero. Use rate() or increase() in PromQL, which both handle counter resets correctly, rather than looking at raw counter values over time.
• Metrics differ between scrapes in a way that doesn't make sense: almost always a multi-worker/multiprocess issue — verify PROMETHEUS_MULTIPROC_DIR is set and shared across all worker processes.

Where to go next

Once this is running, add the standard prometheus-fastapi-instrumentator package for automatic HTTP latency/status histograms so you're not hand-rolling request metrics alongside your custom security ones, and wire the alerting rules above into Alertmanager so cert expiry and auth-failure spikes actually page someone instead of sitting quietly in Grafana.

Mumuni Mohammed

Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.