Kubernetes CreateContainerConfigError: How to Debug and Fix It

What CreateContainerConfigError actually means

CreateContainerConfigError means the kubelet tried to assemble your container's configuration — resolve every environment variable, every envFrom, every projected value — and couldn't, because a ConfigMap or Secret it references doesn't exist, or exists but is missing the key you asked for. The image pulled fine. The process never started. There are no logs, because there was never a container to log.

That's what separates it from its siblings: CrashLoopBackOff means the process ran and died; ImagePullBackOff means the image never arrived. CreateContainerConfigError sits between them — everything downloaded, nothing executed. The kubelet retries on a loop, so the pod sits in this state until the referenced object appears. The good news: of all the pod errors in this series, this one has the most self-explanatory error message. The whole debug is one describe away.

Step 1: Read the event — it names the missing object

Skip the logs (there are none) and go straight to events:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn | tail -15

  Warning  Failed  12s (x8 over 94s)  kubelet
    Error: configmap "payments-config" not found

Or the key-level variant:

  Warning  Failed  9s (x6 over 71s)  kubelet
    Error: couldn't find key DATABASE_URL in ConfigMap prod/payments-config

Or the Secret flavors of both: secret "db-credentials" not found, couldn't find key SMTP_PASSWORD in Secret prod/mail-creds. That message is the entire diagnosis: it tells you which object, which namespace, and — in the key variant — which key. Everything after this step is just figuring out why it's missing.

If you're staring at a fleet, find every affected pod at once:

kubectl get pods -A -o json | jq -r '
  .items[]
  | select(.status.containerStatuses[]?.state.waiting.reason
           == "CreateContainerConfigError")
  | .metadata.namespace + "/" + .metadata.name'

Step 2: Compare what the pod wants with what exists

List every ConfigMap and Secret the pod actually references:

kubectl get pod payments-api-7d9f4c8b6-xk2mn -o json | jq -r '
  .spec.containers[] as $c |
  [ ($c.envFrom[]? | (.configMapRef.name // .secretRef.name)),
    ($c.env[]? .valueFrom? |
      (.configMapKeyRef.name // .secretKeyRef.name)) ]
  | .[] | select(. != null)' | sort -u

Then check each one exists in the pod's namespace — ConfigMaps and Secrets are namespaced, and this is where most "but it exists, I'm looking at it!" cases die:

kubectl get configmap payments-config -n prod
kubectl get secret db-credentials -n prod

For the couldn't find key variant, list the keys the object actually holds:

kubectl get configmap payments-config -n prod -o json | jq '.data | keys'

["DATABASE_HOST", "DATABASE_PORT", "database_url"]

There's the bug: the Deployment asks for DATABASE_URL, the ConfigMap has database_url. Keys are case-sensitive and matched exactly — DB-URL vs DB_URL, a trailing space from a sloppy kubectl edit, or lowercase-vs-uppercase drift between environments all produce this error.

Step 3: Fix the actual cause

The message told you what's missing. These are the five reasons it usually is missing, roughly in order of how often I meet them.

1. Wrong namespace

The ConfigMap exists — in default, while the pod runs in prod. Pods can only reference config objects in their own namespace, full stop. Recreate the object where the pod lives, or better, fix the manifest source so every environment renders its own copy.

2. Key drift between the manifest and the object

The object exists but the key doesn't (the couldn't find key message). Someone renamed a key in the ConfigMap without updating the Deployment, or vice versa. Fix whichever side is wrong — and prefer fixing it in Git, not with kubectl edit, or the next sync reverts you.

3. Deploy ordering: the Deployment landed before its config

The classic GitOps race. The Deployment and its ConfigMap live in different Argo CD apps, or different Helm releases, and the workload synced first. Plain Helm mostly protects you inside a single release (its install order applies ConfigMaps and Secrets before Deployments), but nothing protects you across releases or apps. With Argo CD, pin the order with sync waves:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # on the ConfigMap/Secret

Config objects in wave −1, workloads in wave 0, and the race is gone. If you run Argo CD in production, this annotation belongs on every ConfigMap and Secret that a same-app workload consumes.

4. An operator was supposed to create the Secret — and hasn't

Increasingly common: the Secret isn't in Git at all, it's materialized by External Secrets Operator, Sealed Secrets, or a Vault injector. The pod raced the operator and lost, or the operator itself is failing. Check the intermediate resource, not just the Secret:

kubectl get externalsecret -n prod
kubectl describe externalsecret db-credentials -n prod | tail -8

A SecretSyncedError there (bad SecretStore auth, wrong remote path) is your real root cause — the pod error is downstream noise. Pods usually recover on their own once the operator syncs, because the kubelet keeps retrying.

5. Someone deleted or renamed the object

kubectl delete configmap on the wrong context, a Helm uninstall that took shared config with it, a cleanup script that was too enthusiastic. Check recent history if you have audit logs; recreate from Git if you don't. This failure mode is the argument for keeping every config object declarative and letting nothing be hand-created.

The optional: true escape hatch — use with care

Both key references and whole-object references accept optional: true:

env:
  - name: FEATURE_FLAG_URL
    valueFrom:
      configMapKeyRef:
        name: flags-config
        key: FLAG_URL
        optional: true       # missing key/object → env var simply unset
envFrom:
  - configMapRef:
      name: extra-config
      optional: true

The pod now starts without the object. That's correct for genuinely optional config (feature flags, debug toggles) and dangerous for everything else — you've traded a loud CreateContainerConfigError for an app that boots with an unset DATABASE_URL and fails in some quieter, weirder way at runtime. Default to required; opt into optional deliberately.

Two look-alikes that debug differently

Missing ConfigMap mounted as a volume does not produce CreateContainerConfigError. Volumes are mounted before container config is assembled, so the pod hangs in ContainerCreating with a FailedMount event instead: MountVolume.SetUp failed for volume "config" : configmap "payments-config" not found. Same root cause, different symptom — if your pod is stuck ContainerCreating, check describe for mount events rather than hunting a config error that isn't there.

CreateContainerError (no "Config") is the runtime failing to create the container after config resolved fine: a command pointing at a binary that can't exec, a leftover container with the same name on the node, or a container-runtime fault. The event text comes from the runtime (OCI runtime create failed: ...) rather than the kubelet's config resolver. Read the message the same way — it names the actual failure — but expect the fix to be in your image or command, not your ConfigMaps.

Alert on it — this one loves to hide in rollouts

A CreateContainerConfigError during a rolling update can be nearly invisible: the old ReplicaSet keeps serving, the new pods sit broken, and the Deployment just… never finishes. If you scrape kube-state-metrics in a standard Prometheus + Grafana setup, one rule catches it cluster-wide:

sum by (namespace, pod) (
  kube_pod_container_status_waiting_reason{reason=~"CreateContainerConfigError|CreateContainerError"}
) > 0

Fire it after 5 minutes of persistence — the kubelet's own retries absorb operator-sync races shorter than that, so you only page on the ones that need a human.

Prevent it: validate references before they ship

The whole error class is a referential-integrity bug — a dangling pointer from workload to config — and dangling pointers are checkable in CI. A ~20-line preflight against your rendered manifests catches most of it:

helm template ./chart -f values-prod.yaml > rendered.yaml

# every configMapKeyRef/configMapRef name the workloads reference
refs=$(yq ea '[.spec.template.spec.containers[]?
  | (.envFrom[]?.configMapRef.name,
     .env[]?.valueFrom.configMapKeyRef.name)]
  | flatten | .[] | select(. != null)' rendered.yaml | sort -u)

# every ConfigMap name the same render defines
defined=$(yq ea 'select(.kind == "ConfigMap") | .metadata.name' rendered.yaml | sort -u)

comm -23 <(echo "$refs") <(echo "$defined") | grep . \
  && { echo "dangling ConfigMap refs ^"; exit 1; } || echo "refs OK"

Anything that survives the render check but lives outside the chart (operator-managed Secrets, shared config) is exactly what sync waves and optional: true are for. If you want the guardrail enforced at admission instead of CI, this is a natural fit for policy-as-code with Kyverno — Kyverno can verify a referenced ConfigMap exists via an API-call context and block the workload with a message far friendlier than a stuck pod.

A repeatable checklist

1. kubectl describe pod → the event names the missing object, namespace, and (maybe) key. No logs exist; don't look for them. (Step 1)
2. Diff what the pod references (jq on envFrom + valueFrom) against what exists in that namespace; for key errors, jq '.data | keys' and check case/spelling exactly. (Step 2)
3. Fix the cause, not the symptom: wrong namespace, key drift, GitOps ordering (sync-wave -1 on config), a lagging secrets operator, or a deleted object. (Step 3)
4. Pod stuck ContainerCreating with FailedMount instead? Same root cause via a volume — different symptom. CreateContainerError without "Config"? Runtime problem, check command and the image.
5. Alert on kube_pod_container_status_waiting_reason for both reasons, 5-minute persistence — rollouts hide this error behind a healthy old ReplicaSet.
6. Add a rendered-manifest reference check to CI so the dangling pointer never merges.

Related Reading

• Kubernetes CrashLoopBackOff: How to Debug and Fix It — the next error you'll meet after fixing this one: the container now starts, but the value of that config is wrong and the process dies on boot.
• Kubernetes Pod Stuck in Pending (FailedScheduling): How to Fix It — the error that happens one stage earlier, when the pod can't even land on a node to attempt container creation.
• Kubernetes Pod Evicted: How to Debug and Fix It — the other end of the lifecycle: pods that started fine and were killed by node pressure, and how the kubelet picks its victims.