Build an Alert Tuning Agent: Cut Alert Fatigue by Mining Your Alert History

What This Agent Does

An alert tuning agent attacks alert fatigue at its source: it mines your last 30 days of alert history, ranks every rule by how much pager pain it caused versus how much action it produced, and opens one pull request per noisy alert — a longer for: duration, a saner threshold, a routing downgrade from page to ticket, or (rarely) deletion. The mining and ranking are deterministic PromQL and Python. The LLM enters only where judgment lives: reading the rule, the firing pattern, and the silence history, then proposing which fix and defending it in the PR description. Nothing changes until a human merges.

This is the offline complement to the Alertmanager MCP server, which triages alerts while they fire. Triage treats symptoms; tuning removes the disease. Most teams do this review manually twice a year in a painful spreadsheet meeting, which means eleven and a half months of paging on alerts everyone already ignores. The agent makes it a weekly batch job.

Where Alert History Actually Lives

The awkward truth that shapes this whole design: Alertmanager keeps no durable history. Its API shows you current alerts and silences; yesterday is gone. The history you need is scattered across three places, and the agent reads all of them.

Prometheus itself is the primary source. Every active alert is exported as the synthetic series ALERTS{alertname, alertstate, ...}, and pending-state transitions show up in ALERTS_FOR_STATE. From those two you can reconstruct firing frequency and duration:

# Distinct firings per alert over 30d (approximation: counts
# state transitions; a flapping alert inflates this — which is
# exactly the signal we want to catch)
sum by (alertname) (changes(ALERTS_FOR_STATE[30d]))

# Total firing time per alert: firing samples x eval interval
sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[30d]))

Alertmanager's silences API tells you which alerts humans have already voted against. A rule that has spent 40% of the month inside a silence matcher is a rule the team has tuned socially instead of technically. The agent pulls GET /api/v2/silences, including expired ones still in the retention window, and matches them back to alertnames.

Your pager holds the outcome data. If you page through PagerDuty, the incidents API gives you time-to-acknowledge and whether the incident was resolved with any action or just closed. An alert with a median time-to-ack of four seconds is an alert people dismiss from the lock screen — nobody diagnoses anything in four seconds. That reflex is the measurable signature of fatigue.

Mining the Noise: Scoring Stays in Code

Never ask a model to decide which alerts are noisy — that's arithmetic over history, and it belongs in code, the same division of labor as the burn-rate math in the error budget agent. The miner computes five numbers per alert rule and a composite score:

# noise_miner.py — deterministic ranking, no model involved
import os, httpx

PROM = os.environ["PROM_URL"]
AM = os.environ["ALERTMANAGER_URL"]

def prom(query: str) -> dict[str, float]:
    r = httpx.get(f"{PROM}/api/v1/query", params={"query": query}, timeout=30)
    r.raise_for_status()
    return {s["metric"]["alertname"]: float(s["value"][1])
            for s in r.json()["data"]["result"]}

def mine() -> list[dict]:
    fires = prom('sum by (alertname) (changes(ALERTS_FOR_STATE[30d]))')
    firing_min = prom(
        'sum by (alertname) '
        '(count_over_time(ALERTS{alertstate="firing"}[30d]))')
    # flaps: fired, then resolved again inside ~10 minutes
    flaps = prom(
        'sum by (alertname) (resets((ALERTS{alertstate="firing"} '
        'or vector(0))[30d:5m]))')

    silences = httpx.get(f"{AM}/api/v2/silences", timeout=15).json()
    silenced_min = silenced_minutes_by_alertname(silences)   # match matchers
    ack_stats = pagerduty_ack_seconds()   # {alertname: median_tta_seconds}

    report = []
    for name, n in sorted(fires.items(), key=lambda kv: -kv[1]):
        if n < 5:
            continue          # quiet alerts are not the problem this week
        flap_ratio = flaps.get(name, 0) / n
        score = (
            2.0 * n                              # sheer volume
            + 40.0 * flap_ratio * n              # flapping is worse than volume
            + 0.5 * silenced_min.get(name, 0) / 60
            + (50 if ack_stats.get(name, 999) < 30 else 0)  # reflex-ack
        )
        report.append({
            "alertname": name, "fires_30d": int(n),
            "firing_minutes": int(firing_min.get(name, 0)),
            "flap_ratio": round(flap_ratio, 2),
            "silenced_hours": round(silenced_min.get(name, 0) / 60, 1),
            "median_ack_s": ack_stats.get(name),
            "noise_score": round(score, 1),
        })
    return report[:10]        # the LLM sees the top ten, never the firehose

Two details earn their keep. The n < 5 floor keeps the agent focused on repeat offenders instead of nibbling at rare alerts. And the flap multiplier reflects a real asymmetry: fifty clean firings of one alert might be a bad month, but fifty two-minute flaps are a broken for: clause, and flapping trains on-call humans to ignore the pager faster than anything else.

The LLM's Job: One Diagnosis, One Fix, Forced Schema

For each of the top ten, the agent fetches the actual rule definition from the rules repo and the alert's firing timeline, then makes a single model call with a forced tool schema — the same pattern as every agent in this series:

PROPOSAL_TOOL = {
    "name": "propose_alert_fix",
    "description": "Propose exactly one tuning change for a noisy alert.",
    "input_schema": {
        "type": "object",
        "properties": {
            "diagnosis": {"enum": [
                "threshold_too_tight", "for_too_short", "flapping_signal",
                "should_be_ticket", "duplicate_of_other_alert",
                "dead_alert_delete", "leave_alone"]},
            "proposed_change": {"type": "string",
                "description": "The exact YAML diff to apply, or empty "
                               "for leave_alone."},
            "evidence": {"type": "string",
                "description": "2-4 sentences citing the mined numbers: "
                               "fires, flap ratio, silence hours, ack time."},
            "risk": {"type": "string",
                "description": "What real incident this alert could catch, "
                               "and why the change does not blind us to it."},
        },
        "required": ["diagnosis", "proposed_change", "evidence", "risk"],
    },
}

SYSTEM = (
    "You tune Prometheus alert rules for an SRE team drowning in pages.\n"
    "Prefer the smallest change that kills the noise: bump 'for:' before "
    "touching thresholds, downgrade routing before deleting. Propose "
    "delete ONLY for alerts with zero acknowledged incidents and a "
    "documented duplicate. If the alert looks load-bearing despite the "
    "noise, answer leave_alone — silence is an acceptable output.\n"
    "Never propose changes to alerts tagged slo-burn or deadman.\n"
    "Your proposal becomes a pull request a human reviews. Cite numbers."
)

The risk field is the load-bearing part of the schema. Forcing the model to articulate what the alert protects — before it's allowed to weaken it — is the difference between tuning and vandalism. In testing this style of agent, leave_alone verdicts on genuinely noisy-but-critical alerts are where the model proves it's reasoning rather than pattern-matching "noisy means bad."

Every Change Is a PR With a Test Attached

Proposals ship the way every write in this series ships: as pull requests, one alert per PR, following the GitOps-for-agents pattern. But alert rules have something most agent-touched configs don't: a native unit test harness. Every PR the agent opens must include a promtool test asserting the new behavior, and CI runs it:

# tests/high_latency_p99.test.yaml — shipped in the same PR as the fix
rule_files:
  - ../rules/api-alerts.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'api:latency_p99:5m{app="search-api"}'
        values: '0.9x10 1.4x8 0.9x12'      # an 8-minute spike, then recovery
    alert_rule_test:
      - eval_time: 20m
        alertname: HighLatencyP99
        exp_alerts: []      # after for: 15m, an 8m spike must NOT page
      - eval_time: 20m
        alertname: HighLatencyP99Ticket
        exp_alerts: []      # spike still lands in the ticket queue via rule 2

This turns the review from "does this YAML look right" into "do I agree with this scenario." The reviewer isn't eyeballing a threshold; they're reading an executable statement of the new contract — an eight-minute latency spike no longer wakes anyone — and deciding whether that contract is acceptable. A git revert undoes any tuning decision that turns out wrong, which is more than you can say for the silence-and-forget approach.

Guardrails: The Do-Not-Touch List

Three hard rules live in the wrapper, not the prompt, because prompts are requests and code is law — the same layering argument as the read-only Prometheus MCP server:

• A protected-alerts list. SLO burn-rate alerts, deadman/watchdog alerts, and anything labeled severity: critical with a runbook link are filtered out of the miner's report before the model ever sees them. Burn-rate alerts in particular look noisy to any frequency-based scorer during a bad month — that's them working.
• Rate limiting. Maximum three open tuning PRs at a time. Alert hygiene is a diet, not a purge; twenty simultaneous PRs get rubber-stamped, and a rubber-stamped weakening of your alerting is the worst outcome this agent can produce.
• Deletion requires double evidence. A dead_alert_delete proposal only becomes a PR if the miner independently confirms zero acknowledged pager incidents for that alert in 90 days. The model's opinion alone never deletes a safety net.

Run the first month in shadow mode: proposals land in a review channel as messages, not PRs, and you score them against what the humans decide in the next tuning meeting. The agreement rate tells you when to wire up the PR path.

Honest Limits

The scoring inherits every quirk of its sources. changes(ALERTS_FOR_STATE[30d]) is an approximation — Prometheus restarts and rule reloads perturb it, so treat the fire counts as ranking signal, not audit-grade truth. Silence matching is fuzzy when silences use regex matchers across many alertnames. Ack-time data assumes your pager's incidents map cleanly back to alertnames, which multi-alert grouped notifications muddy. None of this matters much for a top-ten ranking, but don't feed the numbers into anything stricter.

The deeper limit is philosophical: absence of action is not proof of uselessness. An alert that fired 40 times with no action might be noise — or might be the only early warning for a failure mode you've been lucky with. That's why the schema demands a risk statement, why deletion needs double evidence, and why a human merges every change. The agent's real product isn't the YAML diff; it's turning a twice-yearly spreadsheet argument into a weekly, evidence-backed, individually reviewable decision. After a month, your on-call handoffs get shorter for the best possible reason — as the on-call handoff agent would report, there's simply less noise to hand off.