devops

Kubernetes Namespace Stuck in Terminating: How to Debug and Fix It (Finalizers, Broken APIServices, Orphaned CRDs)

Fix a Kubernetes namespace stuck in Terminating: read deletion conditions, fix the broken APIService or leftover finalizer, and use /finalize as a last resort.

October 29, 2026·10 min read·
#kubernetes#sre#devops#containers#reliability#incident-response

What Terminating means for a namespace

A namespace stuck in Terminating is waiting for one of two things: for every object inside it to be gone, or for the cluster to be able to prove that every object inside it is gone. When you run kubectl delete ns, the API server sets deletionTimestamp and the namespace controller inside kube-controller-manager starts a loop: ask discovery for every namespaced resource type the cluster knows about, list each one in the namespace, delete what it finds, and repeat until all lists come back empty. Only then does it remove the kubernetes entry from spec.finalizers, and only then does the object disappear.

That loop fails in exactly four ways, and the namespace tells you which one in status.conditions. Almost every "stuck namespace" blog post skips the conditions and jumps straight to ripping out the finalizer. Do not do that first. The finalizer removal works every time, which is precisely why it is dangerous: it hides a broken APIService, a dead operator, or a cloud resource that is still being billed. Read the conditions, fix the real cause, and keep the raw /finalize call for the one case where it is genuinely the right answer.

Step 1: Read the conditions, not the phase

kubectl get ns staging-payments -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'

A healthy deletion in progress shows all five conditions as False. A stuck one looks like this:

NamespaceDeletionDiscoveryFailure   True   Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1
NamespaceDeletionGroupVersionParsingFailure   False   All legacy kube types successfully parsed
NamespaceDeletionContentFailure   False   All content successfully deleted, may be waiting on finalization
NamespaceContentRemaining   True   Some resources are remaining: certificates.cert-manager.io has 3 resource instances, kafkatopics.kafka.strimzi.io has 12 resource instances
NamespaceFinalizersRemaining   True   Some content in the namespace has finalizers remaining: kafka.strimzi.io/topic-operator in 12 resource instances
Condition that is TrueWhat the controller is telling youGo to
NamespaceDeletionDiscoveryFailureIt cannot enumerate resource types, so it refuses to declare the namespace emptyStep 2
NamespaceDeletionContentFailureA delete call is being rejected, almost always by an admission webhookStep 3
NamespaceContentRemainingObjects still exist inside; the message lists kinds and countsStep 4
NamespaceFinalizersRemainingThose objects carry finalizers their controller is not removingStep 4
All False but still stuckA pod or PVC is in its own slow teardownStep 5

The first condition is the one people miss, because the failing API group usually has nothing to do with the namespace being deleted. A namespace with no metrics in it will still never finish deleting while metrics.k8s.io is unavailable.

Step 2: Discovery failure means a broken APIService

Aggregated APIs register themselves as APIService objects that point at a Service in the cluster. If that backing Service has no healthy endpoints, discovery for that whole group fails, and the namespace controller cannot complete its "list everything" pass for any namespace. One dead metrics-server blocks every namespace deletion in the cluster.

kubectl get apiservice | grep -v 'True'
NAME                                   SERVICE                      AVAILABLE                      AGE
v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (MissingEndpoints)       412d
v1beta1.external.metrics.k8s.io        keda/keda-operator-metrics   False (ServiceNotFound)        91d

The two usual suspects are exactly these: metrics-server with no running pods, and a leftover external-metrics APIService from a KEDA or Prometheus Adapter that was uninstalled with Helm but registered its APIService outside the chart. MissingEndpoints means the Service exists but has no ready pods, so fix the deployment. ServiceNotFound means the Service is gone for good, so the APIService is garbage and can be deleted:

kubectl describe apiservice v1beta1.metrics.k8s.io | sed -n '/Conditions/,$p'
kubectl -n kube-system get pods -l k8s-app=metrics-server
# if the backing service no longer exists anywhere:
kubectl delete apiservice v1beta1.external.metrics.k8s.io

Within a minute the namespace controller retries discovery, the condition flips to False, and in most cases the namespace vanishes on its own. The same broken metrics-server also produces the HPA symptoms covered in Kubernetes HPA shows unknown targets; if you have both problems at once, it is one root cause.

Step 3: Content failure means a webhook is rejecting the delete

NamespaceDeletionContentFailure with a message containing failed calling webhook is a chicken-and-egg problem. A validating or mutating webhook with failurePolicy: Fail is registered for DELETE operations, and its backing pods lived in the namespace you are deleting. The controller deleted the pods first, then tried to delete the next object, the webhook could not be reached, and the policy says fail closed.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json \
  | jq -r '.items[] | .metadata.name as $n | .webhooks[]
           | select(.clientConfig.service.namespace == "staging-payments")
           | "\($n)\t\(.name)\t\(.failurePolicy)\t\(.rules[].operations | join(","))"'

Anything that prints with DELETE or * in the last column and Fail in the third is your blocker. Delete the webhook configuration, since the thing it protected is being removed anyway, and the controller finishes its pass on the next retry. The full diagnosis for webhooks that fail for other reasons, including expired CA bundles and wrong ports, is in how to fix 'failed calling webhook'.

Step 4: Content remaining means finalizers on objects inside

This is the most common case and the one where the fix depends on what is left. First, get the exact list. The condition message gives kinds and counts; this gives names:

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n staging-payments
NAME                                                  READY   SECRET                    AGE
certificate.cert-manager.io/payments-tls              False   payments-tls              201d
NAME                                                  CLUSTER        PARTITIONS   READY
kafkatopic.kafka.strimzi.io/payments-events           prod-kafka     24           True
...

Every object still present has a finalizer that nobody is removing. Check which:

kubectl -n staging-payments get kafkatopic payments-events -o jsonpath='{.metadata.finalizers}'
# ["strimzi.io/topic-operator"]

Now the decision matters, and it is different per finalizer:

FinalizerWho removes itWhat skipping the cleanup actually does
strimzi.io/topic-operatorStrimzi Topic OperatorThe topic stays in Kafka with all its data; the CR is just forgotten
finalizer.managedresource.crossplane.io, finalizer.external.aws.upbound.ioCrossplane providerThe RDS instance, bucket, or VPC keeps running and keeps billing
resources-finalizer.argocd.argoproj.ioArgo CD application controllerArgo stops cascading, child resources in the destination cluster survive
kubernetes.io/pvc-protectionkube-controller-managerNothing dangerous; it only blocks while a pod still mounts the PVC
foregroundDeletionGarbage collectorDependents are already being deleted; wait
An operator you uninstalledNobody, everSafe to remove by hand; the controller no longer exists

The right question is "is the controller that owns this finalizer running?" If the Strimzi operator was scaled to zero, scale it back up and the topics disappear within a minute. If Crossplane is running but the provider pod is crash-looping because its IAM role was deleted along with the environment, fix the credentials first, let it actually deprovision the cloud resources, and then the namespace finishes. Removing a Crossplane finalizer by hand to unstick a namespace is how teams end up paying for an RDS instance for eight months that no manifest references.

When the controller is genuinely gone for good and you have confirmed the external resource is already cleaned up or does not matter, remove the finalizer on the specific object, not on the namespace:

kubectl -n staging-payments patch kafkatopic payments-events --type=merge \
  -p '{"metadata":{"finalizers":null}}'

Do it per object, and do it with the list from the api-resources loop in front of you, so you know what you are orphaning. The same rules apply to a single pod with a stale finalizer, which is covered in depth in Kubernetes pod stuck in Terminating. Helm releases are a special case: if the release's resources are still present because the uninstall itself got stuck, sort that out with the steps in Helm release health: stuck pending-upgrade and failed hooks before touching anything else.

Step 5: Nothing in the conditions but still stuck

If all five conditions are False and the namespace still sits there, the controller is waiting on an object that is itself in a slow teardown. That is almost always a pod past its grace period on a node that is NotReady, or a PVC whose volume detach is hanging. The api-resources loop above will show it. A pod on a dead node needs the node fixed or the pod force-deleted; a PVC with pvc-protection needs the pod that mounts it gone first. Neither is a namespace problem.

The last resort: removing the kubernetes finalizer correctly

Only when the namespace is truly empty, every condition is False, and the object still will not go away, remove the namespace's own finalizer. There is a reason the obvious command does nothing:

kubectl patch ns staging-payments -p '{"metadata":{"finalizers":null}}' --type=merge
# succeeds, changes nothing: the finalizer is in spec.finalizers, not metadata

The kubernetes finalizer lives in spec.finalizers, and the namespace resource ignores edits to that field on the normal endpoint. It is only writable through the /finalize subresource:

kubectl get ns staging-payments -o json \
  | jq 'del(.spec.finalizers)' > /tmp/ns-staging-payments.json
kubectl replace --raw "/api/v1/namespaces/staging-payments/finalize" \
  -f /tmp/ns-staging-payments.json

The namespace disappears instantly. Understand the cost. Any object the controller could not see, because its API group was failing discovery, is still in etcd. Fix that APIService later and the objects reappear, owned by a namespace that no longer exists. Recreate a namespace with the same name and those ghost objects, with their old finalizers, become visible inside the new one, which is the single most confusing failure in this whole area. If you used /finalize to get past a discovery failure, you have not fixed anything; you have deferred the mess to whoever recreates the namespace.

Prevention

  • Uninstall before you delete. helm uninstall and kubectl delete -f the workloads first, let operators do their cleanup while they still exist, then delete the empty namespace. Deleting a namespace that contains the operator and the custom resources it manages is the classic way to create this problem.
  • Keep APIServices healthy or remove them. Add an alert on aggregator_unavailable_apiservice from the kube-apiserver metrics, or simply kubectl get apiservice in your cluster health check. An unavailable APIService breaks more than namespace deletion.
  • Alert on the stuck state itself. kube-state-metrics exposes the phase:
- alert: KubernetesNamespaceStuckTerminating
  expr: kube_namespace_status_phase{phase="Terminating"} == 1
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Namespace {{ $labels.namespace }} has been Terminating for 15 minutes"
    runbook_url: "https://devtocash.com/blog/2026-10-29-kubernetes-namespace-stuck-terminating-fix"

Fifteen minutes is generous enough to cover legitimate slow teardowns of large namespaces and short enough that the person who deleted it is still at their desk.

  • Audit finalizers on cluster-scoped leftovers after every operator uninstall. kubectl get crd -o name | xargs -I{} sh -c 'kubectl get {} -A --no-headers 2>/dev/null | wc -l | xargs echo {}' lists every CRD with the number of surviving instances. A CRD whose operator is gone but which still has instances is a future stuck namespace waiting to happen.

Quick reference

# 1. Which of the four failure modes is it?
kubectl get ns NS -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'
# 2. Discovery failure: find and fix/delete the unavailable APIService
kubectl get apiservice | grep -v True
# 3. Content failure: find webhooks backed by the namespace being deleted
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json | jq '.items[].webhooks[].clientConfig.service'
# 4. Content remaining: list everything left, then fix the owning controller or patch the object
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n NS
# 5. Last resort, only when the namespace is verifiably empty
kubectl get ns NS -o json | jq 'del(.spec.finalizers)' | kubectl replace --raw "/api/v1/namespaces/NS/finalize" -f -

The order is the fix. A namespace stuck in Terminating is the cluster refusing to lie to you about whether something is gone. Make it true, and the namespace leaves on its own.

#kubernetes#sre#devops#containers#reliability#incident-response
D
DevToCashAuthor

Senior DevOps/SRE Engineer · 10+ years · Professional Trader (IDX, Crypto, US Equities)

I write about real infrastructure patterns and trading strategies I use in production and in live markets. No courses, no affiliate hype — just documentation of what actually works.

More about me →