AI Agent Sandbox on Kubernetes: Run LLM-Written Code in gVisor Jobs with No Network and No Credentials

The tool every ops agent eventually asks for

Give an ops agent enough incidents and it will want a run_script tool. It has 40 KB of JSON from a metrics query and wants to compute a percentile. It has two Helm value files and wants a structural diff. Doing arithmetic "in its head" is how you get confidently wrong numbers in a postmortem, so letting the model write a ten-line Python script is the right instinct.

The wrong move is where most harnesses run that script: subprocess.run() inside the agent's own pod. That pod holds the LLM API key, the MCP server tokens, a ServiceAccount token, and network reach to the cloud metadata endpoint and every internal service.

The short answer: run every piece of model-written code in its own ephemeral Kubernetes Job — in a dedicated namespace, under the gVisor RuntimeClass, with no ServiceAccount token, a default-deny NetworkPolicy (DNS included), a non-root read-only filesystem, tight CPU/memory/disk limits, a soft timeout inside the container and a hard one enforced by the control plane, and output capped before it returns to the model. An admission policy makes that shape mandatory, so a bug in your executor can't quietly create an unsandboxed pod. Everything below is the YAML and the ~60 lines of Python that implement it.

Threat model: the script is untrusted input

The model doesn't need to be malicious. It reads logs, PR comments, and alert annotations, and any of those can carry a prompt injection that ends in "now run this snippet." Treat generated code exactly like code a stranger pasted into a web form. There are four things to deny it:

1. Credentials — no ServiceAccount token, no env vars, no mounted Secrets, no cloud metadata.
2. Network — no egress at all. A sandbox that can resolve DNS can exfiltrate data through DNS queries.
3. The host kernel — a plain container shares the node's kernel, so one kernel bug is a node compromise. gVisor puts a user-space kernel between the script and the host.
4. Unbounded resources and time — no infinite loops, memory balloons, disk fills, or 200 parallel executions from a runaway agent loop.

The agent harness post argued that the tool boundary decides what "act" can mean. This is that principle applied to the most dangerous tool you can hand a model: arbitrary code.

Layer 1: a namespace with nothing worth stealing

Anyone who can create pods in a namespace can mount any Secret and use any ServiceAccount in it. So the sandbox namespace must contain nothing: no Secrets, no privileged ServiceAccounts, no other workloads.

apiVersion: v1
kind: Namespace
metadata:
  name: agent-sandbox
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: exec-budget
  namespace: agent-sandbox
spec:
  hard:
    pods: "5"                  # max concurrent executions
    limits.cpu: "5"
    limits.memory: 3Gi
    count/jobs.batch: "30"     # finished Jobs count until TTL removes them

The quota does two jobs. pods: "5" caps concurrency (only non-terminal pods count). count/jobs.batch: "30" combined with a 5-minute Job TTL becomes a crude rate limit: a looping agent burns 30 executions, then gets a 403 from the API server until old Jobs expire. That's backpressure enforced by the platform, not by a prompt.

Layer 2: the gVisor RuntimeClass

gVisor's runsc intercepts the container's syscalls in a user-space kernel, so the script never talks to the host kernel directly. On GKE it's a node-pool flag (gcloud container node-pools create sandbox-pool --sandbox type=gvisor) and the gvisor RuntimeClass already exists. Self-managed and EKS nodes need runsc installed and registered with containerd:

# /etc/containerd/config.toml (containerd 1.x CRI plugin path)
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

Then define the RuntimeClass and pin it to the tainted pool that actually has the handler:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    runtime: gvisor
  tolerations:
    - key: sandbox
      operator: Equal
      value: gvisor
      effect: NoSchedule

The scheduling block matters: if a gVisor pod lands on a node without the runsc handler, it sticks in ContainerCreating with FailedCreatePodSandBox. To confirm you're really sandboxed, run dmesg in a test pod — under gVisor it prints gVisor's own boot messages instead of the node's kernel log. If you can't run gVisor, Kata Containers behind the same RuntimeClass mechanism gives you a microVM boundary instead; the rest of this design is unchanged.

Layer 3: the Job template

apiVersion: batch/v1
kind: Job
metadata:
  name: exec-placeholder
  namespace: agent-sandbox
spec:
  backoffLimit: 0                  # never retry model-written code
  activeDeadlineSeconds: 90        # hard kill, enforced by the control plane
  ttlSecondsAfterFinished: 300     # garbage-collect Job and pod
  template:
    spec:
      runtimeClassName: gvisor
      restartPolicy: Never
      automountServiceAccountToken: false
      enableServiceLinks: false    # no *_SERVICE_HOST env vars
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: exec
          image: registry.internal/agent-exec-py@sha256:<pinned-digest>
          command: ["timeout", "-k", "2", "60", "python3", "-c", "print('hi')"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
              ephemeral-storage: 64Mi
            limits:
              cpu: "1"
              memory: 512Mi
              ephemeral-storage: 128Mi
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 64Mi

The non-obvious choices:

• The script travels as a -c argument, not a ConfigMap. No extra object to create or clean up, no extra RBAC verb for the executor, and the exact code that ran is preserved in the Job spec — which is what your audit trail wants. Linux caps a single argument at about 128 KiB, so the executor rejects scripts over 64 KB.
• Two timeouts, on purpose. When activeDeadlineSeconds fires, the Job controller deletes the pod — and its logs vanish with it. The inner timeout exits with code 124 first, so the pod terminates normally and partial output survives. The outer deadline is the backstop the script can't evade, and it also covers time stuck in scheduling or image pull.
• backoffLimit: 0. Retrying failed model-written code re-runs the same bug. The model should read the traceback and decide what to do next.
• The image is boring and pinned by digest: Python, jq, coreutils. No curl, no kubectl, no cloud CLIs. With no network there's no pip install at runtime — bake the few libraries you need (numpy and pyyaml cover most ops math and parsing) into the image.

Layer 4: default-deny network, DNS included

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: agent-sandbox
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

No egress rules means no egress: no metadata endpoint, no internal services, no DNS. Resist the reflex to "just allow kube-dns" — a resolver is a data channel. This only works if your CNI enforces NetworkPolicy (Calico, Cilium, or the AWS VPC CNI with its policy agent enabled); on a CNI that ignores policies, the manifest applies cleanly and protects nothing. The test matrix below proves it either way.

Make the shape mandatory with an admission policy

The layers above live in a template your executor fills in. A refactor that drops runtimeClassName would silently downgrade every execution to a normal container. A ValidatingAdmissionPolicy (GA since Kubernetes 1.30) turns the template's key properties into law for the namespace:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: agent-sandbox-pod-shape
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
  validations:
    - expression: "has(object.spec.runtimeClassName) && object.spec.runtimeClassName == 'gvisor'"
      message: "agent-sandbox pods must use the gvisor RuntimeClass"
    - expression: "has(object.spec.automountServiceAccountToken) && !object.spec.automountServiceAccountToken"
      message: "agent-sandbox pods must not mount a ServiceAccount token"
    - expression: "!has(object.spec.volumes) || object.spec.volumes.all(v, has(v.emptyDir))"
      message: "agent-sandbox pods may only use emptyDir volumes"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: agent-sandbox-pod-shape
spec:
  policyName: agent-sandbox-pod-shape
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: agent-sandbox

Pod Security restricted already blocks privileged pods, host namespaces, and hostPath. This policy adds the three things PSA doesn't know about: the runtime, the token, and Secret/ConfigMap volumes.

The executor tool

The model sees one tool. The description is part of the guardrail — it tells the model what won't work so it doesn't burn turns trying:

{
  "name": "run_in_sandbox",
  "description": "Run a short Python or bash script in an isolated sandbox. NO network, NO credentials, NO access to the cluster or cloud APIs. Embed input data inline in the script. Returns exit code and the first 16 KB of combined stdout/stderr. Use for calculations, parsing, and diffs only.",
  "input_schema": {
    "type": "object",
    "properties": {
      "script": {"type": "string", "maxLength": 64000},
      "interpreter": {"type": "string", "enum": ["python3", "bash"]},
      "timeout_s": {"type": "integer", "minimum": 5, "maximum": 120}
    },
    "required": ["script", "interpreter"]
  }
}

The harness-side implementation uses the official kubernetes Python client:

import time, uuid, yaml
from kubernetes import client, config
from kubernetes.client.rest import ApiException

NS, MAX_OUTPUT = "agent-sandbox", 16_384
TEMPLATE = open("/etc/executor/job-template.yaml").read()

def run_in_sandbox(script: str, interpreter: str, timeout_s: int = 60) -> dict:
    if interpreter not in ("python3", "bash"):
        return {"status": "rejected", "detail": "interpreter must be python3 or bash"}
    if len(script.encode()) > 64_000:
        return {"status": "rejected", "detail": "script over 64 KB"}
    timeout_s = min(max(timeout_s, 5), 120)

    config.load_incluster_config()
    batch, core = client.BatchV1Api(), client.CoreV1Api()
    name = f"exec-{uuid.uuid4().hex[:10]}"

    job = yaml.safe_load(TEMPLATE)
    job["metadata"]["name"] = name
    job["spec"]["activeDeadlineSeconds"] = timeout_s + 30
    job["spec"]["template"]["spec"]["containers"][0]["command"] = [
        "timeout", "-k", "2", str(timeout_s), interpreter, "-c", script]
    try:
        batch.create_namespaced_job(NS, job)
    except ApiException as e:
        if e.status == 403:  # quota exhausted or admission policy denied
            return {"status": "rejected",
                    "detail": "sandbox budget exhausted; wait before retrying"}
        raise

    give_up = time.time() + timeout_s + 45
    while time.time() < give_up:
        st = batch.read_namespaced_job(name, NS).status
        if st.succeeded or st.failed:
            break
        time.sleep(1)

    pods = core.list_namespaced_pod(
        NS, label_selector=f"batch.kubernetes.io/job-name={name}").items
    if not pods:
        return {"status": "killed", "detail": "hard deadline hit; no output kept"}
    pod = pods[0]
    cs = pod.status.container_statuses or []
    term = cs[0].state.terminated if cs else None
    out = core.read_namespaced_pod_log(
        pod.metadata.name, NS, limit_bytes=MAX_OUTPUT + 1) if term else ""
    return {
        "status": "ok" if term and term.exit_code == 0 else "failed",
        "exit_code": term.exit_code if term else None,
        "reason": term.reason if term else pod.status.reason,  # OOMKilled, Evicted...
        "timed_out": bool(term and term.exit_code == 124),
        "output": out[:MAX_OUTPUT],
        "truncated": len(out) > MAX_OUTPUT,
    }

The executor's own ServiceAccount follows the rules from RBAC for AI agents — a namespaced Role with create and get on jobs, list on pods, and get on pods/log, in agent-sandbox only. No delete: the TTL controller does cleanup, which keeps finished Jobs inspectable for five minutes and keeps the rate-limit quota honest. And note who holds that credential: the harness, never the model and never the sandbox pod.

Prove it: the escape-attempt test matrix

Don't trust the YAML — run hostile scripts through the real tool and assert on the results. This belongs in the same CI suite as your other agent evals:

Two honest notes. The /tmp fill is caught by kubelet eviction, which runs on an interval — the write can briefly overshoot before the pod dies. And fork bombs are not controllable from the pod spec: the per-pod PID cap is the kubelet's podPidsLimit setting, so set it on the sandbox node pool (a few hundred is plenty). The exit-137 case reads the same as any other OOMKilled container — return the reason to the model so it can rewrite the script to stream instead of buffering.

Limits worth knowing before you ship

• Latency. Every execution pays for pod scheduling and sandbox startup, so expect seconds, not milliseconds. Pre-pull the image onto the sandbox pool with a DaemonSet and keep the image small. For chatty workloads needing a long-lived, stateful sandbox with warm pools, watch the kubernetes-sigs/agent-sandbox project — it defines a Sandbox CRD for exactly that, but it's young, so check its current API status before depending on it.
• gVisor compatibility and overhead. Syscall-heavy and I/O-heavy code runs slower under runsc, and some low-level tooling won't run at all. For short parsing and math scripts it's a non-issue.
• No network means no fetching. Data must already be in the agent's context, embedded in the script. That's a feature: the agent fetches data through audited, allow-listed MCP tools, and the sandbox only computes on it.
• The output is still untrusted. A sandbox contains the execution, not the text it returns. If the script processed attacker-controlled logs, its stdout can carry an injection back into the model. Cap it (done), and keep anything that acts on the result behind the same approval gates as every other write.

Takeaway

A code-execution tool makes an ops agent dramatically more accurate, and it is also the single easiest way to turn a prompt injection into a breach. The fix isn't clever prompting; it's boring platform work: an empty namespace, a gVisor RuntimeClass, a locked-down Job template, default-deny egress, two timeouts, a quota that doubles as a rate limit, and an admission policy that makes all of it non-optional. Then run the escape-attempt matrix in CI, so the day someone "simplifies" the template, a test fails instead of a node.