What This Agent Does
A Datadog cost agent finds the custom metrics nobody has queried in 30 days, proves the ones that are still referenced by a dashboard or monitor, estimates how many timeseries a tag configuration would remove before anyone touches it, and drafts log exclusion filters that it has already checked against every log monitor's query. All of the finding and proving is deterministic Python over the Datadog API. The LLM makes exactly one call per candidate, chooses between four fixes, and has to name what breaks. A human applies every change, and every change has a revert path that is one API call long.
Two line items pay for almost every Datadog overage: custom metrics, billed per distinct timeseries above the per-host allotment, and indexed logs, billed per million events. Neither is visible to the team that caused it. A developer who adds a customer_id tag to a histogram sees a nicer dashboard; the invoice sees five percentile series per customer. This is the Datadog sibling of the Prometheus cardinality agent: same discipline, but the knobs are different and some of them are one-way doors.
The Four Signals, All From the API
Everything the agent needs is in five endpoints. Scope one application key to usage_read, metrics_read, monitors_read, and dashboards_read for the read half, and a second key to metrics_metadata_write and logs_write_exclusion_filters for the write half. The write key is never in the LLM's environment.
export DD_SITE=api.datadoghq.com # or api.datadoghq.eu, api.us5.datadoghq.com
H=(-H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY_RO")
# 1. Which custom metrics cost the most? Average distinct timeseries per hour, last month.
curl -s "${H[@]}" "https://$DD_SITE/api/v1/usage/top_avg_metrics?month=2026-10-01&limit=500" \
| jq '.usage[] | select(.metric_category=="custom")
| {metric_name, avg_metric_hour, max_metric_hour}' | head -40
# 2. Which metrics has anyone queried in the last 30 days (the maximum window)?
curl -s "${H[@]}" "https://$DD_SITE/api/v2/metrics?filter[queried]=true&window[seconds]=2592000" \
| jq -r '.data[].id' > queried.txt
# 3. Which dashboards, monitors, notebooks, and SLOs reference one metric?
curl -s "${H[@]}" "https://$DD_SITE/api/v2/metrics/checkout.request.latency/assets" \
| jq '.data.attributes | {dashboards: (.dashboards|length), monitors: (.monitors|length),
notebooks: (.notebooks|length), slos: (.slos|length)}'
# 4. How many timeseries would this metric produce if only these tags were kept?
curl -s "${H[@]}" "https://$DD_SITE/api/v2/metrics/checkout.request.latency/estimate?filter[groups]=env,service,status_code&filter[hours_ago]=49" \
| jq '.data.attributes.estimated_output_series'
The set difference between signal one and signal two is the candidate pool: expensive, and not queried by anyone for a month. Signal three is the safety net, because "not queried" and "not referenced" are different claims. A monitor that has not fired evaluates its query every minute, so it shows up in the queried list, but a dashboard nobody opened in October does not, and that dashboard may be the one the on-call engineer opens during the next incident. Signal four is the number that makes a tag change reviewable: the estimate endpoint returns what the metric would cost after the change, straight from Datadog's own cardinality math, so the PR quotes a real figure rather than a model's guess.
For logs the equivalent is one aggregate query over indexed events, which is exactly the quantity you pay for:
curl -s "${H[@]}" -X POST "https://$DD_SITE/api/v2/logs/analytics/aggregate" \
-H "Content-Type: application/json" -d '{
"compute": [{"aggregation": "count", "type": "total"}],
"filter": {"from": "now-7d", "to": "now", "query": "*"},
"group_by": [{"facet": "service", "limit": 30}, {"facet": "status", "limit": 8}]
}' | jq '.data.buckets | sort_by(-.computes.c0)[:15] | .[] | {by, n: .computes.c0}'
On most accounts the top of that list is a handful of services emitting info or debug lines on every request, plus health-check access logs. Those are indexed, searchable, and retained for fifteen days, and almost nobody searches them.
The Miner: Ranking Stays in Code
The ranking is arithmetic and belongs in a script that produces the same answer every Monday. It also applies the protected list before any metric is shown to a model:
# dd_cost_miner.py — deterministic, no model involved
import os, httpx
DD = f"https://{os.environ['DD_SITE']}"
H = {"DD-API-KEY": os.environ["DD_API_KEY"],
"DD-APPLICATION-KEY": os.environ["DD_APP_KEY_RO"]}
PROTECTED_PREFIXES = ("slo.", "sli.", "trace.", "runtime.") # never shown to the model
MONTH = "2026-10-01"
def top_custom() -> dict[str, float]:
r = httpx.get(f"{DD}/api/v1/usage/top_avg_metrics", headers=H,
params={"month": MONTH, "limit": 500}, timeout=60)
r.raise_for_status()
return {u["metric_name"]: u["avg_metric_hour"] for u in r.json()["usage"]
if u["metric_category"] == "custom"}
def queried_30d() -> set[str]:
r = httpx.get(f"{DD}/api/v2/metrics", headers=H, timeout=60,
params={"filter[queried]": "true", "window[seconds]": 2592000})
r.raise_for_status()
return {m["id"] for m in r.json()["data"]}
def assets(metric: str) -> dict[str, int]:
r = httpx.get(f"{DD}/api/v2/metrics/{metric}/assets", headers=H, timeout=60)
a = r.json()["data"]["attributes"]
return {k: len(a.get(k, [])) for k in ("dashboards", "monitors", "notebooks", "slos")}
def mine() -> list[dict]:
cost, queried = top_custom(), queried_30d()
total = sum(cost.values()) or 1
out = []
for name, series in sorted(cost.items(), key=lambda kv: -kv[1])[:60]:
if name.startswith(PROTECTED_PREFIXES) or series / total < 0.005:
continue
refs = assets(name)
out.append({
"metric": name,
"avg_timeseries": round(series),
"share": round(series / total, 3),
"queried_30d": name in queried,
"refs": refs,
"hard_unused": name not in queried and sum(refs.values()) == 0,
})
return out[:15]
The hard_unused flag is the only thing that permits a deletion-class proposal later, and it requires both signals to agree. A metric that is unqueried but has one monitor is a tag-cardinality candidate, not a removal candidate.
The LLM's Job: One Fix, Forced Schema, Name What Breaks
For each candidate the agent passes the miner row, the metric's current tag list from the all-tags endpoint, the titles of the referencing dashboards and monitors, and one estimate result for a proposed tag set. Then it makes a single call:
PROPOSAL_TOOL = {
"name": "propose_datadog_fix",
"description": "Propose exactly one cost change for one Datadog custom metric.",
"input_schema": {
"type": "object",
"properties": {
"diagnosis": {"enum": [
"configure_tags", # Metrics without Limits: keep only listed tags
"drop_percentiles", # distribution: stop paying 5x for p50..p99
"stop_emitting_unused", # hard_unused only: PR to the exporter/app
"leave_alone"]},
"keep_tags": {"type": "array", "items": {"type": "string"},
"description": "Tags to retain. Must be a subset of the current tag list."},
"estimated_timeseries_after": {"type": "integer"},
"evidence": {"type": "string",
"description": "2-4 sentences citing avg_timeseries, share, the "
"queried flag, and the asset counts."},
"what_breaks": {"type": "string",
"description": "Which dashboard widget or monitor groups by a dropped "
"tag, and why that is acceptable. 'nothing' is suspicious."},
},
"required": ["diagnosis", "keep_tags", "estimated_timeseries_after",
"evidence", "what_breaks"],
},
}
SYSTEM = (
"You reduce Datadog custom-metric spend for an SRE team.\n"
"A metric that any monitor or SLO references may only receive configure_tags "
"with every tag the monitor groups by kept, or leave_alone.\n"
"stop_emitting_unused is allowed only when hard_unused is true, and it "
"produces a pull request to the emitting service, not an API call.\n"
"High-cardinality tags like customer_id, request_id, container_id, or "
"pod_name are almost never worth a timeseries each. Say so.\n"
"Your proposal is reviewed by a human. Quote the estimate, not a guess."
)
Two Datadog-specific rules shape the enum. First, a tag configuration is not a drop. Once a metric has one, any query grouped by a tag outside the kept set returns nothing, so a dashboard widget that breaks down latency by status_code goes blank the moment status_code leaves the list. That is why keep_tags must be a subset of what exists and why the wrapper cross-checks every kept list against the group-by tags it extracts from referencing monitor queries. Second, distributions are the expensive case. Percentile aggregations cost five extra timeseries per tag combination, and many teams enabled them for a p99 panel they built once. The drop_percentiles branch sets include_percentiles to false, and the wrapper refuses it if any monitor query contains p95 or p99 on that metric.
Applying a Tag Configuration, With the Estimate on the Diff
The wrapper, not the model, holds the write key and makes the call. It fetches the volumes endpoint before and after, so the saving is measured rather than asserted:
def configure_tags(metric: str, keep: list[str], metric_type: str,
include_percentiles: bool = False) -> dict:
body = {"data": {"type": "manage_tags", "id": metric, "attributes": {
"metric_type": metric_type, # gauge | count | rate | distribution
"tags": keep,
**({"include_percentiles": include_percentiles}
if metric_type == "distribution" else {}),
}}}
r = httpx.post(f"{DD}/api/v2/metrics/{metric}/tags",
headers=H_RW, json=body, timeout=60)
if r.status_code == 409: # configuration already exists: update it
r = httpx.patch(f"{DD}/api/v2/metrics/{metric}/tags",
headers=H_RW, json=body, timeout=60)
r.raise_for_status()
return r.json()
def revert(metric: str) -> None: # the whole undo path
httpx.delete(f"{DD}/api/v2/metrics/{metric}/tags", headers=H_RW, timeout=60)
For a count or rate metric the default aggregation after configuration is sum over space and time, and for a gauge it is avg over both. If a dashboard widget was using max on a gauge, say for a queue-depth panel, that aggregation disappears unless you add it to the aggregations list in the same request. The wrapper scans referencing dashboard JSON for the aggregator used on the metric and adds any non-default one, which is the kind of detail a model asked only for a tag list will not volunteer.
The change ships as a Slack approval request in the shape the human-in-the-loop post describes: metric, current average timeseries, estimated timeseries after, the dashboards and monitors it checked by title, and the one-line revert. Nothing is applied on a Friday.
Log Exclusion Filters That Cannot Starve a Monitor
Exclusion filters are where Datadog cost agents hurt people, because the index configuration endpoint replaces the whole filter list on every update. A naive PUT with one new filter silently deletes the three filters a colleague added last year. The wrapper always reads, appends, and writes the full object:
def add_exclusion(index: str, name: str, query: str, sample_rate: float) -> dict:
cur = httpx.get(f"{DD}/api/v1/logs/config/indexes/{index}", headers=H_RW).json()
filters = cur.get("exclusion_filters", [])
if any(f["name"] == name for f in filters):
raise RuntimeError(f"exclusion {name} already exists on {index}")
filters.append({"name": name, "is_enabled": True,
"filter": {"query": query, "sample_rate": sample_rate}})
body = {"filter": cur["filter"], "exclusion_filters": filters,
"num_retention_days": cur.get("num_retention_days")}
r = httpx.put(f"{DD}/api/v1/logs/config/indexes/{index}", headers=H_RW, json=body)
r.raise_for_status()
return r.json()
Before any filter is proposed, the wrapper pulls every log monitor and tests overlap with a count query, because excluded logs are not available to log monitors:
def monitor_overlap(exclusion_query: str) -> list[tuple[str, int]]:
mons = httpx.get(f"{DD}/api/v1/monitor", headers=H,
params={"type": "log alert"}).json()
hits = []
for m in mons:
# query looks like: logs("service:checkout status:error").index("main")...
search = m["query"].split('logs("', 1)[1].split('")', 1)[0]
n = httpx.post(f"{DD}/api/v2/logs/analytics/aggregate", headers=H, json={
"compute": [{"aggregation": "count", "type": "total"}],
"filter": {"from": "now-7d", "to": "now",
"query": f"({search}) AND ({exclusion_query})"},
}).json()["data"]["buckets"]
count = n[0]["computes"]["c0"] if n else 0
if count:
hits.append((m["name"], count))
return hits
A non-zero overlap does not kill the proposal; it changes its shape. Metrics generated from logs are computed at ingestion, before exclusion, so the fix is to create a log-based metric for the monitor's condition, re-point the monitor at it, and only then add the exclusion. The LLM's schema for logs has a convert_monitor_first diagnosis for exactly this case. Its other two options are sampling (keep 10% of info lines from a chatty service so the Explorer still shows the shape of normal traffic) and full exclusion for health checks and load-balancer probes, which nobody has ever searched for on purpose. The same retention-versus-ingestion distinction drives the CloudWatch Logs cost agent; here it is sharper, because excluded logs are still ingested and still billed at the ingestion rate. Exclusion only cuts the indexing line.
Guardrails That Live in the Wrapper
- Two keys. The model's process only ever sees the read-only application key. The write key is loaded by the applier after a human approves, and it is scoped to metadata and exclusion filters, nothing else.
- Assets are a hard filter. Any
stop_emitting_unusedverdict on a metric with a non-zero asset count is rejected before it is shown to anyone, whatever the evidence says. - Subset rule.
keep_tagsmust be a subset of the metric's current tags and a superset of every tag any referencing monitor groups by. Both checks are code. - Budget. Five metric changes and two exclusion filters per week, with an action budget that demotes the agent to read-only when a revert is called twice in a month, the pattern from the circuit breaker post.
- Measured, not claimed. Seven days after each change the wrapper re-reads the volumes endpoint and the index's daily count and posts the before and after to the same thread. A change whose measured saving is under half of its estimate is reverted automatically and flagged for a human to understand.
Run it in shadow mode for the first month: proposals land in a channel with their estimates, and the team marks each one as "would have done this" or not. The log side in particular tends to find a service nobody realised was logging every request body, and that conversation is better had by a person.
Honest Limits
The queried-metrics window is capped at 30 days, so a metric that only matters during the quarterly close or an annual load test looks unused in any single run. Keep a small allow-list for seasonal metrics and union three monthly runs before trusting a hard-unused verdict. The assets endpoint covers dashboards, monitors, notebooks, and SLOs; it does not see a query a teammate types into Metrics Explorer, a Terraform-managed dashboard that has not been applied yet, or an external tool hitting the query API with its own key. The estimate endpoint uses the last 49 hours of data by default, so a tag that is only present during batch windows will be under-counted, and the wrapper should run it at a peak hour as well.
The deeper limit is the asymmetry. A tag configuration saves money the moment it lands and breaks a dashboard the moment somebody needs it, which may be weeks later. That is why the revert is one DELETE, why what_breaks must be non-empty, why the write key is never near the model, and why a human approves every change. The product is not the API calls. It is replacing the yearly "why did Datadog go up 35%" meeting with a weekly list of individually reviewable, individually revertable decisions, each with Datadog's own estimate attached.