The MCP Gateway Pattern: One Front Door for All Your Ops AI Agents

The N-times-M problem nobody warns you about

An MCP gateway is a single service that sits between your AI agents and all of your ops MCP servers. Agents connect to one endpoint; the gateway routes tool calls to the right upstream server, enforces a per-agent tool allowlist, holds all the upstream credentials, and writes one audit log for everything. If you run more than one MCP server and more than one agent, it is the difference between a system you can reason about and a pile of point-to-point connections you can't.

Here's how you get into trouble. You build a read-only kubectl MCP server. It works, so you add a PromQL server, a Loki log search server, and an Alertmanager triage server. Then a second agent shows up — the CI triage bot wants logs and metrics but must never see alert silencing. Now you have 4 servers times 2 agents = 8 connection configs, each with its own credentials, its own allowlist logic (or none), and its own log file. Add a third agent and you're maintaining twelve. Every new server multiplies against every agent.

The fix is the same one we've used for twenty years of service architecture: put a gateway in front.

What the gateway owns (and what it doesn't)

Four responsibilities belong in the gateway, and they're all cross-cutting:

• Aggregation and namespacing. One endpoint exposes k8s_get_pod_logs, prom_range_query, loki_search, am_list_alerts. The agent sees one coherent toolbox; prefixes tell you (and the model) which system a tool touches.
• Identity and allowlists. Each agent authenticates to the gateway with its own token. The gateway decides which tools that identity may call. The on-call agent gets all four systems; the CI bot gets logs and metrics only.
• Credential custody. Upstream tokens — the Kubernetes ServiceAccount, the Prometheus basic-auth pair, the Loki tenant header — live only in the gateway's environment. Agents never hold infra credentials, which shrinks the leak surface the way secrets management for AI agents says you should: the model can't exfiltrate what it never had.
• One audit log and one kill switch. Every call from every agent lands in a single structured log, and disabling a misbehaving tool is one config change instead of a hunt across servers.

Just as important is what the gateway must not do: domain logic. Query cost caps, result summarization, and PromQL step floors belong in the upstream servers, next to the systems they protect. The gateway routes, authorizes, and records. Keep it thin or it becomes a second place where every team's rules live.

Build it: a FastMCP proxy with middleware

FastMCP 2.x can mount remote MCP servers behind prefixes, and its middleware hook sees every tool call — which is exactly the interception point a gateway needs. The whole thing is under a hundred lines.

# gateway.py — MCP gateway: aggregation, allowlists, audit, kill switch
import json
import os
import time
import yaml
from fastmcp import FastMCP
from fastmcp.server.dependencies import get_http_headers
from fastmcp.server.middleware import Middleware, MiddlewareContext

UPSTREAMS = {
    "k8s":  "http://k8s-mcp.agents.svc:8000/mcp",
    "prom": "http://prom-mcp.agents.svc:8000/mcp",
    "loki": "http://loki-mcp.agents.svc:8000/mcp",
    "am":   "http://alertmanager-mcp.agents.svc:8000/mcp",
}

gateway = FastMCP("ops-gateway")

for prefix, url in UPSTREAMS.items():
    # Mounted tools appear as <prefix>_<toolname>, e.g. prom_range_query.
    gateway.mount(FastMCP.as_proxy(url), prefix=prefix)

The policy file is boring on purpose — reviewable in a pull request, no code required to change who may call what:

# policy.yaml — per-agent tool allowlists + global kill switch
agents:
  oncall-agent:
    token_env: ONCALL_AGENT_TOKEN
    allow:
      - "k8s_*"
      - "prom_*"
      - "loki_*"
      - "am_*"
  ci-triage-bot:
    token_env: CI_BOT_TOKEN
    allow:
      - "loki_search"
      - "prom_range_query"
      - "prom_instant_query"

disabled_tools: []        # add "am_create_silence" here to kill it everywhere

Middleware does the enforcement. Every tool call — regardless of which upstream it targets — passes through one function:

import fnmatch

POLICY = yaml.safe_load(open("policy.yaml"))
TOKENS = {
    os.environ[a["token_env"]]: name
    for name, a in POLICY["agents"].items()
}

class GatePolicy(Middleware):
    async def on_call_tool(self, ctx: MiddlewareContext, call_next):
        headers = get_http_headers()
        token = headers.get("authorization", "").removeprefix("Bearer ").strip()
        agent = TOKENS.get(token)
        tool = ctx.message.name

        if agent is None:
            raise PermissionError("unknown agent identity")
        if tool in POLICY["disabled_tools"]:
            raise PermissionError(f"{tool} is globally disabled")
        allowed = POLICY["agents"][agent]["allow"]
        if not any(fnmatch.fnmatch(tool, pat) for pat in allowed):
            raise PermissionError(f"{agent} may not call {tool}")

        start = time.monotonic()
        try:
            result = await call_next(ctx)
            outcome = "ok"
            return result
        except Exception:
            outcome = "error"
            raise
        finally:
            print(json.dumps({
                "ts": int(time.time()), "agent": agent, "tool": tool,
                "args": ctx.message.arguments, "outcome": outcome,
                "ms": round((time.monotonic() - start) * 1000),
            }), flush=True)

gateway.add_middleware(GatePolicy())

if __name__ == "__main__":
    gateway.run(transport="http", host="0.0.0.0", port=9000)

Note the shape of the audit line: agent, tool, arguments, outcome, latency. When your security team asks "what exactly can the CI bot touch, and what did it do last Tuesday?", the answer is one YAML file and one jq query — not an archaeology dig across four services.

Wiring an agent to it

The agent's own config collapses to a single entry. Claude Code, for example:

{
  "mcpServers": {
    "ops": {
      "type": "http",
      "url": "https://mcp-gateway.internal:9000/mcp",
      "headers": { "Authorization": "Bearer ${ONCALL_AGENT_TOKEN}" }
    }
  }
}

One URL, one token, and the tool list the model sees is exactly what the allowlist permits — undiscoverable tools can't be hallucinated into use. Compare that with the pre-gateway world where every agent host carried four server URLs and four credential sets.

Operational details that bite

Rate limit per agent, not globally. A runaway loop in one agent shouldn't starve the on-call agent mid-incident. A token-bucket keyed on agent identity inside the middleware (say, 30 calls/minute for the CI bot, more for on-call) turns a runaway agent from an outage into a log line. Runaway tool-call loops are the number-one failure mode we flagged in observability for DevOps AI agents, and the gateway is the natural choke point to both see and stop them.

The gateway is a SPOF — treat it like one. You've concentrated all agent access into one process. That's the point, but it means the gateway deserves the same treatment as any tier-1 internal service: two replicas behind a Service, liveness probes, and its audit stream shipped somewhere durable. The upside of the SPOF is symmetric: one deployment to scale, patch, and monitor, and pulling the plug on all agent access during a security incident is kubectl scale --replicas=0.

Latency is real but small. You're adding one HTTP hop, typically single-digit milliseconds on a cluster network. Against LLM inference measured in seconds per turn, it's noise. If a tool is latency-critical enough that a hop matters, it probably shouldn't be behind an LLM at all.

Tool-list churn. When an upstream adds a tool, it appears through the gateway automatically — but denied by default until you extend an allowlist. That default is correct. New capability reaching agents should be a reviewed policy change, the same least-privilege posture argued in the agent harness as infrastructure.

When you don't need this

One agent talking to one MCP server needs no gateway — the server's own guardrails are enough, and an extra service is pure overhead. The threshold is the multiplication: the moment you have two agents with different trust levels, or you catch yourself copying the same auth snippet into a third server, the cross-cutting concerns have outgrown the point-to-point wiring. Build the gateway that week, while the migration is still four config edits — not after agent number five ships with a credential it shouldn't have.

The pattern compounds from there. Central identity makes per-agent budgets enforceable, the unified audit log is the raw material for agent evals and incident review, and approval gates for write-path tools have an obvious place to live. The gateway isn't just plumbing — it's where your agent platform stops being a collection of experiments and starts being infrastructure.