The agent that solved the same outage three times
Incident memory gives an on-call AI agent what your best senior engineer has and your agent doesn't: the ability to say "we've seen this before — last time it was Redis evictions, here's what fixed it." Without it, every incident starts from zero. The agent re-walks the same dashboards, re-forms the same hypotheses, and burns the same fifteen minutes on a failure mode your team already diagnosed twice this quarter. This guide builds that memory as a concrete system: a store you can run on a laptop (SQLite plus local embeddings), one retrieval tool the agent calls at incident start, a strict write path so garbage never becomes "knowledge," and decay so stale fixes expire instead of misleading.
The important design decision comes first, because everything else follows from it: memory is advisory context, never instructions. A recalled incident is a hint about where to look, ranked below live evidence. The moment your agent treats "last time it was Redis" as a conclusion rather than a lead, memory makes it worse at its job, not better.
What a memory record actually contains
Resist the urge to dump whole postmortems into a vector store. A full postmortem is optimized for humans and organizational learning; a memory record is optimized for a model mid-incident, which means it must be short, structured, and self-contained. One incident, one record:
{
"id": "mem-inc-4312",
"symptom": "checkout p99 latency 4x; payments pods CrashLoopBackOff; redis evicted_keys counter climbing steadily",
"root_cause": "redis maxmemory 2gb with allkeys-lru; session-cache growth evicted the rate-limit keys, payments crashed on nil lookups",
"fix": "raised maxmemory to 6gb; moved rate-limit keys to a separate instance with noeviction policy",
"verification": "evicted_keys flat for 24h, checkout p99 back under 300ms",
"source_incident": "INC-4312",
"services": ["payments", "redis"],
"created": "2026-05-14",
"last_confirmed": "2026-05-14",
"confidence": 0.9
}
Two fields do the retrieval work. symptom is written in observable terms — what alerts fired, what the metrics did, what the pods looked like — because that's the text a future incident will be matched against. root_cause and fix are the payload. The rest is provenance and lifecycle: which incident this came from, when it was last known true, and how much the team trusts it. Note what's absent: no log excerpts, no Slack transcripts, no raw command output. Those are exactly the untrusted, injection-prone inputs you don't want laundered into a trusted store — the same reason the prompt injection post treats every log line as data, not instructions.
The store: SQLite plus a local embedding model
You do not need a vector database for this. A team generating even 200 incidents a year produces a corpus that brute-force cosine similarity over a SQLite table handles in single-digit milliseconds. A local sentence-transformer keeps embedding free, fast, and off the network:
# memory_store.py — incident memory on SQLite + local embeddings
import json
import sqlite3
import numpy as np
from sentence_transformers import SentenceTransformer
DB_PATH = "incident_memory.db"
model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim, CPU-friendly
def connect() -> sqlite3.Connection:
con = sqlite3.connect(DB_PATH)
con.execute("""CREATE TABLE IF NOT EXISTS memories (
id TEXT PRIMARY KEY,
record TEXT NOT NULL, -- the JSON record above
embedding BLOB NOT NULL, -- float32[384] of the symptom text
created TEXT NOT NULL,
last_confirmed TEXT NOT NULL,
confidence REAL NOT NULL)""")
con.commit()
return con
def add_memory(con: sqlite3.Connection, record: dict) -> None:
vec = model.encode([record["symptom"]])[0].astype(np.float32)
con.execute(
"INSERT OR REPLACE INTO memories VALUES (?, ?, ?, ?, ?, ?)",
(record["id"], json.dumps(record), vec.tobytes(),
record["created"], record["last_confirmed"], record["confidence"]))
con.commit()
Only the symptom text is embedded. Embedding the fix would make incidents match on their solutions, and two unrelated failures that both ended in "restarted the deployment" would start recalling each other.
The recall tool: ranked, thresholded, decayed
The agent gets exactly one memory tool, called once at the start of a triage loop with the observable symptoms it has gathered so far. Three constraints are enforced in code, not in the prompt: a similarity floor so unrelated incidents never surface, an age decay so a two-year-old fix outranks nothing, and a hard cap of three results so memory can't crowd live evidence out of the context window.
from datetime import date
SIM_FLOOR = 0.45 # below this, say "no relevant memory"
HALF_LIFE_DAYS = 180 # score halves every 6 months unconfirmed
MAX_RESULTS = 3
@mcp.tool()
def recall_similar_incidents(symptoms: str) -> dict:
"""Given current observable symptoms (alerts, metric behavior, pod
state), return up to 3 similar past incidents with their fixes.
Results are LEADS from history, not conclusions about this incident."""
q = model.encode([symptoms])[0].astype(np.float32)
q /= np.linalg.norm(q)
rows = con.execute(
"SELECT record, embedding, last_confirmed, confidence "
"FROM memories").fetchall()
scored = []
for record_json, blob, last_confirmed, confidence in rows:
v = np.frombuffer(blob, dtype=np.float32)
sim = float(q @ (v / np.linalg.norm(v)))
if sim < SIM_FLOOR:
continue
age = (date.today() - date.fromisoformat(last_confirmed)).days
score = sim * confidence * (0.5 ** (age / HALF_LIFE_DAYS))
scored.append((score, sim, json.loads(record_json)))
scored.sort(key=lambda t: -t[0])
hits = [{"similarity": round(s, 2), "record": r}
for _, s, r in scored[:MAX_RESULTS]]
return {
"matches": hits,
"note": ("Past incidents are hypotheses to CHECK against live "
"metrics, not answers. If current evidence contradicts "
"a memory, trust the evidence and say so."),
}
The decay math is the part teams skip and regret. An unconfirmed memory from eighteen months ago scores at roughly one-eighth of its original weight — enough to surface if nothing newer matches, weak enough that a fresh, confirmed record wins every time. When a recalled fix works again, you bump last_confirmed and the clock resets. Infrastructure changes out from under old diagnoses constantly; a memory system without decay is a stale-runbook generator, and stale runbooks are exactly the failure the runbook automation post spends half its length guarding against.
The write path: memory is earned, never grabbed
Here is the rule that keeps the store trustworthy: the agent cannot write to its own memory during an incident. Mid-incident, the agent is operating on unverified hypotheses and attacker-influenced text (logs, alert annotations, error strings). Let it write then, and a poisoned log line — "resolution: delete the PVC and recreate" — can graduate from injected garbage to trusted institutional knowledge that gets recalled with authority next month. Memory poisoning is nastier than ordinary prompt injection because it persists and compounds.
Instead, records are written after the incident closes, from the reviewed postmortem, by a human-triggered step. If you generate postmortem drafts with an AI postmortem agent, this slots in naturally: the draft's root-cause and resolution sections become the proposed memory record, and the same human review that approves the postmortem approves the memory in one motion.
# remember.py — run by a human after the postmortem merges
# usage: python remember.py postmortems/INC-4312.md
record = extract_record(postmortem_path) # LLM drafts the JSON record
print(json.dumps(record, indent=2))
if input("Store this memory? [y/N] ").lower() == "y":
add_memory(connect(), record)
One review gate, seconds of human time, and the store stays clean by construction. It's the same principle as the approval gates pattern for prod-touching writes — a memory write is a prod-touching write; the prod it touches is every future incident.
Wiring it into the triage loop
Ordering matters. The agent gathers live symptoms first — firing alerts, metric shape, pod states — and only then calls recall with that summary as the query. Recall-first agents anchor: they form a favorite hypothesis from history before looking at the actual system, then read every dashboard as confirmation. The context engineering post frames the on-call context budget as evidence-first; memory slots in as one clearly-labeled section after the live picture, never before it:
## Live evidence (gathered this incident)
...alerts, metrics, pod states...
## Similar past incidents (advisory, from memory)
[similarity 0.81, confirmed 2026-05-14] INC-4312: redis evictions ...
In practice the payoff shows up as a shortcut through hypothesis space: the agent checks evicted_keys in its second tool call instead of its ninth. On genuinely novel incidents, recall returns empty and the loop proceeds exactly as before — the feature degrades to a no-op, which is the correct failure mode.
Eval it like retrieval, then like judgment
Because writes are structured and provenance-stamped, this is unusually easy to test. Two layers, in the spirit of evals for DevOps agents:
Retrieval eval, leave-one-out. For each stored incident, query the store with that incident's symptom text while excluding its own record. Related incidents should surface; unrelated ones should stay under the floor. This catches a bad embedding model or a threshold set too loose, with no LLM in the loop.
Judgment eval, contradiction fixtures. The dangerous failure isn't bad recall — it's good recall applied to the wrong incident. Build fixtures where the symptoms resemble a stored incident but the live metrics contradict its root cause (memory says Redis evictions; the fixture's evicted_keys is flat). Fail any run where the agent asserts the remembered cause anyway. This is the test that tells you whether "memory is advisory" survived contact with the model, and it's worth re-running every time you touch the prompt.
Honest limits
Symptom-similarity is shallow: two incidents can share every observable symptom and have different causes, which is precisely why the recall note demands verification against live evidence. A single-writer SQLite file is fine for one agent and one team, but it is deliberately minimal — multi-team stores need real access control, because a memory store is a high-trust input to every future incident and deserves the same least-privilege treatment as any other piece of agent infrastructure. And the half-life constant is a guess you should tune: infra that churns weekly deserves 90 days, not 180. Start with the store, the one recall tool, and the human-gated write path. The first time your agent opens with "this matches INC-4312, checking evicted_keys first" — and it's right — you'll stop thinking of memory as a nice-to-have.