Secrets Management for DevOps AI Agents: Credentials Without Leaks

An agent with credentials is a new kind of leak surface

A DevOps AI agent is only useful if it can authenticate: to the Kubernetes API, to AWS, to Grafana, to GitHub. The rule that makes this safe is simple to state and easy to violate: the agent process holds credentials; the model never sees them. Tokens live in the tool layer — environment variables, mounted files, a Vault lease — and every credential is short-lived and scoped to the narrowest role that does the job. Any secret that does enter the context window must be treated as leaked, because context is copied into transcripts, telemetry, and the model provider's API.

That last part is what makes agents different from every service you've secured before. A normal workload reads a secret and uses it. An agent runtime records everything — and then helpfully replays it in three places you probably haven't threat-modeled:

1. The context window itself. If a tool result contains a token, the model can quote it in a summary, paste it into a PR description, or echo it into a Slack message. Models repeat what they read; that's the job.
2. Transcripts and traces. Session logs are the whole point of agent observability — but a trace that captures raw tool output captures every secret that passed through it, forever, in your logging backend.
3. The API boundary. Everything in context is sent off-box to the model provider on every turn. A secret in the prompt is a secret that has left your infrastructure.

So the goal is not "store secrets carefully." It is "arrange the tool boundary so secrets never cross it." Here is how to do that with the tools you already run.

Rule 1: Credentials live in the tool layer, not the prompt

The worst pattern in the wild is also the most common one: an API key pasted into the system prompt ("use this token when calling the Grafana API"), or an agent with a generic shell and a readable .env file. Both put the credential one cat away from the context window.

Instead, the tool — an MCP server or a typed wrapper — authenticates on the agent's behalf and returns data, never the credential it used:

import os, requests

GRAFANA_TOKEN = os.environ["GRAFANA_SA_TOKEN"]  # held by the process

def query_dashboard(uid: str) -> dict:
    r = requests.get(
        f"https://grafana.internal/api/dashboards/uid/{uid}",
        headers={"Authorization": f"Bearer {GRAFANA_TOKEN}"},
        timeout=10,
    )
    r.raise_for_status()
    body = r.json()
    # Return the payload the model needs — never headers, never config
    return {"title": body["dashboard"]["title"],
            "panels": [p["title"] for p in body["dashboard"]["panels"]]}

The model calls query_dashboard("k8s-cost") and gets panel titles. It cannot ask for the token because no tool returns it. Two corollaries that people miss:

• Don't give a secrets-holding agent a generic file-read or shell tool. If the agent can run env or read ~/.aws/credentials, rule 1 is fiction. Scope the tool surface the way you'd scope its harness permissions — the two controls only work together.
• Pass subprocesses a clean environment. A tool that shells out should hand the child an explicit allowlist of variables, not the parent's full env, so a debugging command can't dump AWS_SECRET_ACCESS_KEY into a tool result by accident.

Rule 2: Short-lived and scoped, or it isn't safe to automate

A static admin token in an agent's environment is a standing incident. The fix is the same one FinOps and security teams already use for human access: dynamic, expiring credentials bound to a minimal role.

Vault dynamic secrets are the cleanest fit for database and cloud access. The agent's tool requests a credential at call time and gets one that expires on its own:

# Role that mints read-only Postgres creds with a 15-minute TTL
vault write database/roles/agent-readonly \
    db_name=analytics \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD \
      '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl=15m max_ttl=1h

# The tool layer calls this per session — never the model
vault read database/creds/agent-readonly

If the lease leaks into a transcript, it is dead within fifteen minutes and it could only ever SELECT. Compare that to the blast radius of a leaked root password and the engineering effort pays for itself the first time something goes wrong.

On Kubernetes, use projected ServiceAccount tokens instead of long-lived secrets. The kubelet rotates them, and you pin the audience and lifetime:

apiVersion: v1
kind: Pod
metadata:
  name: oncall-agent
spec:
  serviceAccountName: agent-readonly
  containers:
    - name: agent
      image: ghcr.io/acme/oncall-agent:1.4
      volumeMounts:
        - name: sa-token
          mountPath: /var/run/secrets/tokens
  volumes:
    - name: sa-token
      projected:
        sources:
          - serviceAccountToken:
              path: agent-token
              expirationSeconds: 900   # 15 minutes, kubelet-rotated
              audience: kubernetes

Bind agent-readonly to a Role with only get, list, and watch on the resources the agent diagnoses — the RBAC discipline is identical to scoping any service, except the client is chattier and less predictable, so the floor matters more. A leaked 15-minute read-only token is an annoyance; a leaked cluster-admin kubeconfig is a resume update.

Rule 3: Prefer identity over secrets entirely

The best secret is the one that never exists. For agents running inside CI — a failure-triage agent or a Terraform plan reviewer — OIDC workload identity replaces stored cloud keys completely:

permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/agent-plan-reader
      aws-region: us-east-1
      # No aws-access-key-id. No stored secret to leak or rotate.

GitHub signs a short-lived identity token for the run; AWS trusts the federation and issues temporary credentials scoped to agent-plan-reader. There is no long-lived key in any secret store, so there is nothing for a transcript to capture and nothing to rotate after an incident. If you're inspecting one of these federation tokens to debug a trust policy, they're ordinary JWTs — decode the payload and check the aud and sub claims against your condition keys.

The same pattern applies on EKS (IRSA / Pod Identity), GKE (Workload Identity), and Azure (federated credentials). Anywhere an agent runs on a platform with an identity provider, federation beats a stored key.

Rule 4: Redact at the tool boundary, because secrets will show up anyway

Even with perfect credential hygiene, secrets arrive in data. A pod crashes because someone put a connection string in an env var, and your agent runs the diagnostic that prints it. kubectl describe pod happily displays env values; log lines contain bearer tokens; Terraform state contains passwords in plaintext. The tool layer needs an output filter — deterministic code, not a polite request to the model:

import re

PATTERNS = [
    re.compile(r"(?i)(api[_-]?key|token|passwd|password|secret)\s*[=:]\s*\S+"),
    re.compile(r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]+"),  # JWT
    re.compile(r"AKIA[0-9A-Z]{16}"),          # AWS access key ID
    re.compile(r"xox[bpars]-[0-9A-Za-z-]+"),  # Slack tokens
    re.compile(r"ghp_[0-9A-Za-z]{36}"),       # GitHub PAT
]

def scrub(tool_output: str) -> str:
    for p in PATTERNS:
        tool_output = p.sub("[REDACTED]", tool_output)
    return tool_output

Run every tool result through scrub before it reaches the model, and the model's summaries, transcripts, and traces are clean by construction. This is the same philosophy as defending against prompt injection: the model's behavior is probabilistic, so the controls you rely on must sit outside the model, in code that always runs. Regexes won't catch every exotic secret — they don't need to. They need to catch the 95% of leaks that are boring, and cost you one function.

For Kubernetes specifically, prefer purpose-built read tools that omit env values and secret data by default over raw kubectl get -o yaml, which serializes everything, stringData included.

Rule 5: Assume a leak happened, and make that survivable

Finally, close the loop the way you would for any credential system:

• Scan transcripts like you scan git. Run gitleaks or trufflehog over session logs on a schedule. An agent transcript is a repo full of machine-written text; treat it with the same suspicion.

gitleaks detect --no-git --source /var/log/agent-sessions/ \
    --report-format json --report-path /tmp/agent-leaks.json

• Rotate on sight. If a real secret appears in any transcript, it went off-box — rotate it immediately, don't debate exploitability. Short TTLs from rule 2 make this cheap; with 15-minute leases, "rotate" often means "wait."
• Audit credential use, not just agent actions. Vault's audit log and CloudTrail tell you what each short-lived credential actually did — which is also your evidence trail when an approval-gated mutation goes through and someone asks what the agent touched.

The checklist

• The model never receives a credential: no keys in prompts, no tools that return tokens, no generic shell or file access alongside held secrets.
• Every credential is short-lived (minutes to an hour) and scoped to a read-mostly role; Vault dynamic secrets or projected ServiceAccount tokens over static keys.
• Wherever a platform identity exists — CI, EKS, GKE — use OIDC federation and store nothing.
• Every tool output passes through a deterministic redaction filter before reaching the context window.
• Transcripts get secret-scanned, and anything found is rotated without discussion.

None of this is exotic. It's the secrets discipline you already apply to services, tightened for a client that copies everything it reads into places you don't fully control. Get it right and handing an agent real access stops being the scary part of the project.