Kubernetes 'failed calling webhook' (Internal error occurred): How to Debug and Fix Admission Webhook Failures

What the error actually means

Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred:
failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook:
Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s":
dial tcp 10.96.141.20:443: connect: connection refused

This error does not come from your manifest and it does not come from the controller you were trying to configure. It comes from the API server. Before persisting your object it had to call an admission webhook registered in a ValidatingWebhookConfiguration or MutatingWebhookConfiguration, the HTTPS call failed, and because that webhook's failurePolicy is Fail, the API server rejected your request rather than skip the check. Nothing is wrong with the Ingress. Something is wrong with the pod that was supposed to validate it, or with the path between the API server and that pod.

The same failure shows up in three disguises. kubectl apply prints it directly. Helm prints it as a failed release with the webhook text buried in the message. And the one that hides best: a Deployment that says 0/3 with no pods at all, where kubectl describe rs shows FailedCreate ... Internal error occurred: failed calling webhook, because a mutating webhook on pods is down and the ReplicaSet controller cannot create anything. Argo CD surfaces it as a sync error with the identical text.

The tail of the message tells you which of five problems you have. This post walks them in the order you should check.

Step 1: Find the webhook configuration behind the name

The quoted name in the error (validate.nginx.ingress.kubernetes.io, webhook.cert-manager.io, validate.kyverno.svc-fail) is the webhooks[].name inside a configuration object, not the object's own name. Search both kinds:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -o json | jq -r '.items[] | .kind + "\t" + .metadata.name + "\t" + (.webhooks[].name)' \
  | grep validate.nginx

ValidatingWebhookConfiguration	ingress-nginx-admission	validate.nginx.ingress.kubernetes.io

Now read what it points at and how it is allowed to fail:

kubectl get validatingwebhookconfiguration ingress-nginx-admission -o json \
  | jq '.webhooks[] | {name, service: .clientConfig.service, failurePolicy, timeoutSeconds, namespaceSelector, rules: [.rules[].resources]}'

{
  "name": "validate.nginx.ingress.kubernetes.io",
  "service": { "name": "ingress-nginx-controller-admission", "namespace": "ingress-nginx", "path": "/networking/v1/ingresses", "port": 443 },
  "failurePolicy": "Fail",
  "timeoutSeconds": 10,
  "namespaceSelector": null,
  "rules": [["ingresses"]]
}

Four fields matter. service is where the API server will connect. failurePolicy: Fail is why you are blocked instead of warned. timeoutSeconds (default 10, maximum 30) is how long it waits. And namespaceSelector: null means this webhook fires for every namespace, which becomes important in Step 6.

Step 2: connection refused and no endpoints: the backing pod is down

The API server does not route through the Service's ClusterIP. It resolves the Service to a ready endpoint and connects to the pod IP on the target port. So the first check is whether any ready endpoint exists:

kubectl get endpointslices -n ingress-nginx -l kubernetes.io/service-name=ingress-nginx-controller-admission
kubectl get pods -n ingress-nginx -l app.kubernetes.io/component=controller
kubectl get svc -n ingress-nginx ingress-nginx-controller-admission -o jsonpath='{.spec.ports[0].targetPort}'; echo

If the pod list is empty, scaled to zero, Pending, or in CrashLoopBackOff, you have found it. Common ways an admission pod disappears without anyone noticing:

• A node drain or spot interruption took the only replica and the replacement is Pending on resources. Most Helm charts default to one replica for the controller that also serves the webhook.
• Someone scaled the controller to zero to "stop traffic" during an incident, forgetting it also validates every Ingress in the cluster.
• The controller is up but the port is wrong. ingress-nginx serves the webhook on container port 8443, cert-manager on 10250, Kyverno on 9443. A Service whose targetPort was edited, or a chart upgrade that renamed the port, gives connection refused against a perfectly healthy pod. Compare the Service targetPort to the pod's containerPort and to the --validating-webhook flag in the controller's args.

The fix is whatever brings a ready pod back: fix scheduling, roll back the bad image, or scale up. The webhook recovers the moment the EndpointSlice has an address. Nothing needs to be re-registered.

Step 3: service "..." not found: the orphaned webhook

failed calling webhook "webhook.cert-manager.io": failed to call webhook:
Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s":
service "cert-manager-webhook" not found

ValidatingWebhookConfiguration and MutatingWebhookConfiguration are cluster-scoped. Deleting a namespace removes the Deployment, the Service, and the Secret, but leaves the webhook configuration behind, still pointing at a Service that no longer exists. From then on every create or update matching its rules fails cluster-wide. This is the number one cause of "I uninstalled cert-manager and now I cannot create Ingresses in a different namespace."

It also happens after helm uninstall when the webhook configuration was created by a Helm hook Job rather than as a tracked release resource, and after an operator CRD cleanup that mirrors the finalizer problem covered in Pod Stuck in Terminating.

List every webhook whose Service does not exist:

for kind in validatingwebhookconfigurations mutatingwebhookconfigurations; do
  kubectl get $kind -o json | jq -r --arg k "$kind" \
    '.items[] | . as $c | .webhooks[] | select(.clientConfig.service != null)
     | "\($k)\t\($c.metadata.name)\t\(.clientConfig.service.namespace)/\(.clientConfig.service.name)"'
done | while IFS=$'\t' read -r kind cfg svc; do
  kubectl get svc -n "${svc%/*}" "${svc#*/}" >/dev/null 2>&1 || echo "ORPHAN  $kind  $cfg  -> $svc"
done

ORPHAN  validatingwebhookconfigurations  cert-manager-webhook  -> cert-manager/cert-manager-webhook
ORPHAN  mutatingwebhookconfigurations    cert-manager-webhook  -> cert-manager/cert-manager-webhook

If the product is genuinely gone, delete the configurations:

kubectl delete validatingwebhookconfiguration cert-manager-webhook
kubectl delete mutatingwebhookconfiguration cert-manager-webhook

If you intend to reinstall, do the delete first anyway. A reinstall that itself has to pass through the broken webhook can fail on the same error, which is the deadlock in Step 6.

Step 4: context deadline exceeded: the API server cannot reach the pod

Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s":
context deadline exceeded

The pod is running, the endpoint is listed, and the API server still times out. On a self-managed cluster this is rare. On managed clusters it is the most common case, because the control plane lives outside your node network and has to cross a firewall to reach the pod.

GKE private clusters. By default the control plane may reach nodes only on ports 443 and 10250. Every other webhook port must be opened explicitly, and the port that matters is the pod's target port, not the Service port:

MASTER_CIDR=$(gcloud container clusters describe prod --region us-central1 \
  --format='value(privateClusterConfig.masterIpv4CidrBlock)')
gcloud compute firewall-rules create gke-prod-master-to-webhooks \
  --network prod-vpc --direction INGRESS \
  --source-ranges "$MASTER_CIDR" \
  --target-tags gke-prod-node \
  --allow tcp:8443,tcp:9443

cert-manager works out of the box on GKE precisely because it chose port 10250, which is already open for the kubelet.

EKS. The control plane ENIs sit in your VPC and use the cluster security group. If your node groups use a custom security group, it needs an inbound rule from the cluster security group on the webhook target port. The default managed node group configuration shares the cluster SG and does not hit this.

NetworkPolicy on the webhook namespace. A default-deny ingress policy in ingress-nginx or kyverno blocks the API server like any other client. Traffic from the control plane does not carry a pod selector, so you have to allow it by CIDR. On GKE that is the master CIDR above; on EKS it is the VPC CIDR, or the control plane ENI addresses if you want to be tight. If you generate policies from observed flows, as in the NetworkPolicy agent post, the API server's calls to webhook pods are exactly the flow that quiet observation windows miss.

The webhook is just slow. A policy engine that calls out to an image registry or an external service on every admission can exceed timeoutSeconds under load. The API server metric for this is unambiguous:

histogram_quantile(0.99, sum by (le, name) (
  rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])
))

A webhook whose p99 is near its timeoutSeconds will time out intermittently and look like a flaky network. Raise the timeout toward 30 only as a stopgap, since every second spent there is added latency on every kubectl apply in the cluster.

You can prove the network path is the problem rather than the pod by testing from a pod in another namespace. A 400 or 405 response means the server is reachable and the path works; only the API server's path is blocked:

kubectl run -n default webhook-probe --rm -it --restart=Never --image=curlimages/curl:8.10.1 -- \
  curl -sk -o /dev/null -w '%{http_code}\n' \
  https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses

Step 5: x509 errors: the caBundle is stale

failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s":
tls: failed to verify certificate: x509: certificate signed by unknown authority

The API server verifies the webhook's serving certificate against clientConfig.caBundle inside the webhook configuration. Three mechanisms keep those two in sync, and each breaks in its own way:

• ingress-nginx runs two Helm hook Jobs: ingress-nginx-admission-create generates the cert into Secret ingress-nginx-admission, and ingress-nginx-admission-patch writes the CA into the webhook configuration. If someone deleted and recreated the Secret, or the patch Job failed on a node with no image access, the two drift apart. Check kubectl get jobs -n ingress-nginx for a non-complete patch Job.
• cert-manager uses its cainjector and the annotation cert-manager.io/inject-ca-from on the webhook configuration. If cainjector is down, a renewed serving cert is never copied into caBundle. The same renewal-tracking that catches expiring certs in the TLS certificate expiry agent should include cainjector's health.
• Kyverno manages its own certificates and rewrites its webhook configurations itself. A stale bundle usually means Kyverno was restored from a backup or migrated between clusters while the generated Secret kyverno-svc.kyverno.svc.kyverno-tls-ca was not.

Compare what the API server trusts with what the pod serves:

kubectl get validatingwebhookconfiguration ingress-nginx-admission \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d \
  | openssl x509 -noout -subject -enddate

kubectl get secret -n ingress-nginx ingress-nginx-admission -o jsonpath='{.data.ca}' \
  | base64 -d | openssl x509 -noout -subject -enddate

Different subjects or serials mean drift. For ingress-nginx the clean fix is to let the hooks run again: helm upgrade with the same values re-executes both Jobs. For a one-off repair, patch the bundle directly:

CA=$(kubectl get secret -n ingress-nginx ingress-nginx-admission -o jsonpath='{.data.ca}')
kubectl patch validatingwebhookconfiguration ingress-nginx-admission --type=json \
  -p "[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",\"value\":\"$CA\"}]"

For Kyverno, restarting the admission controller Deployment forces it to regenerate and re-register.

Step 6: The deadlock, and how to get out of it safely

The nastiest version: a webhook whose rules match pods in all namespaces, including its own. The webhook pod dies. The ReplicaSet controller tries to create a replacement pod. The API server calls the webhook to admit that pod. The webhook is down. The pod is rejected. Forever.

You will see it as a Deployment stuck at 0/1 with FailedCreate events in kubectl describe rs, and every other Deployment in the cluster stuck the same way. Kyverno ships a namespaceSelector excluding kyverno and kube-system to avoid exactly this, and Gatekeeper honors the admission.gatekeeper.sh/ignore namespace label. Home-grown webhooks almost never think of it.

Getting out requires temporarily removing the webhook from the request path. Two options, in order of preference:

Flip to fail-open. Objects are admitted without the check, and the API server emits a warning you can see in audit logs and in apiserver_admission_webhook_fail_open_count:

kubectl patch mutatingwebhookconfiguration my-injector --type=json \
  -p '[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

Delete the configuration. Kyverno recreates its own on startup, so deleting kyverno-resource-validating-webhook-cfg is safe and self-healing. ingress-nginx's is recreated by the Helm hook on the next upgrade. For anything else, save it first:

kubectl get validatingwebhookconfiguration my-policy -o yaml > my-policy-webhook.bak.yaml
kubectl delete validatingwebhookconfiguration my-policy

Be deliberate about what you just switched off. If the webhook was Kyverno enforcing image signatures or a policy-as-code guardrail, every pod admitted during the window skipped it. Treat the window as an incident with a start and end time, and re-enable before closing it.

Hardening so this does not page you again

• Two replicas and a PodDisruptionBudget for every Deployment that serves a failurePolicy: Fail webhook. One replica is the default in most charts and the single biggest cause of Step 2.
• priorityClassName: system-cluster-critical on admission pods, so they schedule ahead of the workloads they gate during node pressure.
• A namespaceSelector on every webhook, at minimum excluding kube-system and the webhook's own namespace. Also scope rules to the resources you actually inspect; a validating webhook for Ingresses has no business matching pods.
• Choose failurePolicy per webhook. Security controls stay Fail. Mutating conveniences like default labels or sidecar injection can usually be Ignore with an alert, because silently missing a label beats a cluster-wide create outage.
• Alert on the API server's view, which catches every variant above before a human runs kubectl apply:

groups:
  - name: admission-webhooks
    rules:
      - alert: AdmissionWebhookCallFailures
        expr: |
          sum by (name) (
            rate(apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error"}[5m])
          ) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API server cannot call admission webhook {{ $labels.name }}"
          description: "Every create/update matching this webhook is being rejected. Check its endpoints, caBundle, and control-plane firewall."

error_type="calling_webhook_error" is specifically the API server failing to reach or trust the webhook. Rejections the webhook itself issued show up as no_error with a rejection_code of 400 or 403, which is the webhook doing its job. Feed the first into your event triage as a cluster-wide finding, not a per-namespace one, because that is what it is.

Quick reference

# 1. Which configuration owns the webhook name in the error?
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json \
  | jq -r '.items[] | .metadata.name + "\t" + (.webhooks[].name)'

# 2. Does its Service have a ready endpoint on the right target port?
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>

# 3. Does the Service exist at all? (orphan check)
kubectl get svc -n <ns> <svc>

# 4. Is the API server reaching it? (timeouts = firewall/NetworkPolicy/slow webhook)
#    GKE: open master CIDR -> node target port. Check webhook p99 duration.

# 5. Does caBundle match the served cert?
kubectl get validatingwebhookconfiguration <cfg> -o jsonpath='{.webhooks[0].clientConfig.caBundle}' \
  | base64 -d | openssl x509 -noout -subject -enddate

# 6. Emergency: fail open, then fix, then fail closed again
kubectl patch validatingwebhookconfiguration <cfg> --type=json \
  -p '[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

The error says Internal error occurred and names your manifest, which sends most people hunting in the wrong file. Read the URL and the tail of the message instead. They name the component that is actually broken, and one of the five checks above fixes it.