Why an agent needs a query layer, not a scrape endpoint
When an on-call AI agent tries to answer "why is checkout latency up?", the worst thing you can do is hand it a shell and your Prometheus URL. It will curl the raw HTTP API, guess at label names, fire an unbounded range query across two weeks of high-cardinality series, and either time out or knock over the instance everyone else is querying during the incident. The agent doesn't need Prometheus the server — it needs a bounded, structured PromQL surface with guardrails baked in.
That surface is exactly what the Model Context Protocol (MCP) is for. An MCP server exposes a small set of typed tools; the agent calls them with validated arguments and gets structured results back. This is the same read-only discipline behind the safe kubectl MCP server — the design principle is identical: give the agent a narrow door, not the whole building. Here we build the Prometheus equivalent: an agent can ask questions about your metrics, and it structurally cannot mutate, delete, or overload anything.
The threat model in one paragraph
Read-only does not mean harmless. A PromQL query is code that runs on your monitoring server. Three things can go wrong even without a write path: a query so expensive it degrades Prometheus for real users; a range + step combination that returns millions of samples and blows up the agent's context (and your token bill); and label values leaking data the agent shouldn't summarize. An MCP server is the right place to enforce all three, because every query has to pass through your tool code before it reaches the TSDB.
The tool surface: three tools, no more
Resist the urge to expose everything. An agent answering reliability questions needs to run an instant query, run a range query, and discover what metrics exist. That's it.
# server.py — a read-only Prometheus MCP server built on FastMCP
import os
import time
import httpx
from fastmcp import FastMCP
PROM_URL = os.environ["PROM_URL"] # e.g. http://prometheus:9090
MAX_RANGE_SECONDS = 6 * 3600 # never span more than 6h
MIN_STEP_SECONDS = 15 # floor the resolution
MAX_POINTS = 720 # hard cap on returned samples
HTTP_TIMEOUT = 10.0
mcp = FastMCP("prometheus-readonly")
client = httpx.Client(base_url=PROM_URL, timeout=HTTP_TIMEOUT)
Everything the agent can do is one of the three functions below. Because there is no /api/v1/admin call, no remote-write, and no delete-series tool anywhere in the file, "read-only" is a property of the code, not a policy you hope holds.
Tool 1: instant query, with a query-shape guard
BANNED = ("/api/v1/admin", "delete_series", "clean_tombstones")
def _reject_if_dangerous(promql: str) -> None:
lowered = promql.lower()
if any(b in lowered for b in BANNED):
raise ValueError("query references an admin/write endpoint")
if len(promql) > 2048:
raise ValueError("query too long — refine it")
@mcp.tool()
def instant_query(promql: str) -> dict:
"""Run an instant PromQL query and return the current value(s)."""
_reject_if_dangerous(promql)
r = client.get("/api/v1/query", params={"query": promql})
r.raise_for_status()
return _summarize(r.json())
Tool 2: range query, where the real guardrails live
This is where an unbounded agent does the most damage. You compute the step from the requested window so the point count can never exceed MAX_POINTS, and you refuse windows longer than your ceiling. The agent asks for a time range in plain seconds; your code decides the resolution.
@mcp.tool()
def range_query(promql: str, lookback_seconds: int) -> dict:
"""Run a range query over the last N seconds. Step is computed, not caller-controlled."""
_reject_if_dangerous(promql)
if lookback_seconds > MAX_RANGE_SECONDS:
raise ValueError(f"lookback capped at {MAX_RANGE_SECONDS}s")
end = int(time.time())
start = end - lookback_seconds
# Floor the step so we never return more than MAX_POINTS samples per series.
step = max(MIN_STEP_SECONDS, lookback_seconds // MAX_POINTS)
r = client.get("/api/v1/query_range", params={
"query": promql, "start": start, "end": end, "step": step,
})
r.raise_for_status()
return _summarize(r.json(), computed_step=step)
Note what the caller cannot set: the step. A common failure mode is an agent asking for a 6-hour window at a 1-second step — that's 21,600 points per series, times however many series the selector matches. By deriving step from the window and flooring it, the worst case is bounded no matter what the model requests.
Tool 3: metric discovery, so the agent stops guessing
Half of an agent's bad queries come from inventing metric names. Give it a cheap way to look them up, and it will ground its PromQL in reality instead of hallucinating http_request_duration_seconds when your metric is actually http_server_request_duration_seconds.
@mcp.tool()
def list_metrics(prefix: str = "") -> list[str]:
"""Return metric names, optionally filtered by prefix. Cheap metadata call."""
r = client.get("/api/v1/label/__name__/values")
r.raise_for_status()
names = r.json().get("data", [])
names = [n for n in names if n.startswith(prefix)] if prefix else names
return names[:200] # never dump the full cardinality list
Summarize results — don't dump raw samples into context
The single biggest cost saver is refusing to return raw JSON. A range result with 700 points across 40 series is a wall of numbers that helps neither the model nor your token bill. Summarize each series to the shape an incident actually needs: first, last, min, max, and the count.
def _summarize(payload: dict, computed_step: int | None = None) -> dict:
result = payload.get("data", {}).get("result", [])
out = []
for series in result[:50]: # cap series returned
metric = series.get("metric", {})
values = series.get("values") or [series.get("value")]
nums = [float(v[1]) for v in values if v]
if not nums:
continue
out.append({
"labels": metric,
"first": nums[0], "last": nums[-1],
"min": min(nums), "max": max(nums), "points": len(nums),
})
resp = {"series": out, "truncated": len(result) > 50}
if computed_step:
resp["step_seconds"] = computed_step
return resp
For the metric selectors themselves, keep them ordinary PromQL — something like rate(http_requests_total{code=~"5.."}[5m]) passes through untouched. The server never rewrites the query; it only bounds how far and how finely it runs, and how much comes back.
Harden the Prometheus side too
The MCP server is your application-layer fence, but defense in depth means the TSDB shouldn't trust it blindly. Run Prometheus itself with the write and admin surfaces disabled so a bug in your tool code can't escalate:
prometheus \
--web.enable-lifecycle=false \
--web.enable-admin-api=false \
--storage.tsdb.retention.time=15d \
--query.max-samples=50000000 \
--query.timeout=10s
--query.max-samples and --query.timeout are the ones that matter here: even if a malformed query slips past your point cap, Prometheus will refuse to load an absurd number of samples and will kill anything that runs too long. That is the backstop that protects the humans querying the same instance during an incident.
Don't skip the eval
An agent with query access is only useful if its queries are right, and only safe if the guardrails actually hold. Before this goes anywhere near a real incident, run it against a fixed set of scenarios — "find the service with the highest error rate", "is memory trending up on the payments pods" — and check both the answer and the query it generated. That is the whole argument in evals for DevOps AI agents: you test the tool-using behavior, not just the model. Pair it with the least-privilege thinking from the agent harness as infrastructure and treat the MCP server as a component you version and monitor like any other.
Where this fits
A read-only PromQL MCP server is the metrics half of an on-call agent's senses; the tracing half is covered in observability for DevOps AI agents, and the human-facing dashboards it complements are the ones from the production Prometheus + Grafana setup. The pattern generalizes: pick the smallest tool surface that answers the question, compute the expensive parameters instead of trusting the caller, summarize before returning, and disable the write path in two places, not one. Do that, and an agent can help you read your metrics during an incident without ever becoming a risk to them.