Kubernetes 'exec format error': How to Debug and Fix It (arm64 vs amd64 Images, Missing Shebangs, QEMU)

What the error actually means

$ kubectl logs api-7c9d4b6f5-x2k8q
exec /docker-entrypoint.sh: exec format error

$ kubectl describe pod api-7c9d4b6f5-x2k8q
    Last State:  Terminated
      Reason:    StartError
      Message:   failed to create containerd task: failed to create shim task:
                 OCI runtime create failed: runc create failed: unable to start
                 container process: exec /docker-entrypoint.sh: exec format error: unknown
      Exit Code: 128

exec format error is the kernel's ENOEXEC. The container runtime asked Linux to execute the entrypoint, and the kernel looked at the first bytes of the file and refused: this is not a program I know how to run on this CPU. Nothing in your application ran. There are no application logs because the process never started, which is why the only line in kubectl logs is the error itself.

In Kubernetes this surfaces as a CrashLoopBackOff with Reason: StartError and exit code 128 on containerd nodes, and it never heals on its own because the file and the CPU are not going to change between restarts. In roughly nine cases out of ten the cause is a CPU architecture mismatch: an arm64 image scheduled onto an amd64 node, or the reverse. The other cases are a text script with a broken or missing shebang, or a cross-platform build on a CI runner with no QEMU registered. The rest of this post walks the checks in the order that finds the cause fastest.

If you are still at the pull stage rather than the start stage, you are looking at a different error. See the ImagePullBackOff guide for no match for platform in manifest, which is the same root cause caught one step earlier.

Step 1: Compare the node's architecture with the image's

Two facts settle most investigations. Which CPU is the node, and which CPU was the image built for.

# where did the pod land, and what is that node?
NODE=$(kubectl get pod api-7c9d4b6f5-x2k8q -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture,KERNEL:.status.nodeInfo.kernelVersion

NAME                           ARCH    KERNEL
ip-10-0-41-17.ec2.internal     arm64   6.1.102-108.177.amzn2023.aarch64

Now the image. crane (from go-containerregistry) and docker buildx imagetools both read the registry without pulling layers:

IMAGE=registry.example.com/platform/api:1.42.0

# multi-arch image: lists one manifest per platform
crane manifest "$IMAGE" | jq -r '.manifests[]? | "\(.platform.os)/\(.platform.architecture)"'

# single-arch image: the line above prints nothing, so read the config instead
crane config "$IMAGE" | jq -r '"\(.os)/\(.architecture)"'

linux/amd64

An amd64-only image on an arm64 node. That is your answer. If you prefer Docker tooling, docker buildx imagetools inspect "$IMAGE" prints the same list under Manifests: with a Platform: line each.

Two things trip people up when reading manifests. First, images built with BuildKit's default provenance attestations show extra entries with platform unknown/unknown. They are attestation blobs, not runnable images, and they are harmless. Second, docker image inspect --format '{{.Architecture}}' on your laptop tells you about the copy you pulled, which Docker already selected for your laptop's CPU. Always inspect the registry, not the local cache.

Step 2: Find out how the mismatch happened

Knowing why the wrong architecture got built or scheduled decides which fix is right.

The image was built on a developer laptop. Since Apple Silicon became the default engineering laptop, docker build with no --platform flag produces a linux/arm64 image. Push that to a registry, deploy to a fleet of x86 nodes, and every pod crashes with this error. Check the config's architecture as above, and check the pipeline: if the image tag was pushed by a human rather than CI, this is almost certainly it.

The cluster gained a second architecture. The other direction is now just as common. Someone added Graviton instances to an EKS node group for the price advantage, or a Karpenter NodePool was left with no kubernetes.io/arch requirement so it happily provisioned arm64 capacity when it was cheapest. Your amd64-only images worked for months and then started crashing on exactly the pods that landed on the new nodes. Confirm with:

kubectl get nodes -L kubernetes.io/arch,karpenter.sh/nodepool,eks.amazonaws.com/nodegroup

If the ARCH column shows a mix, every single-arch image in the cluster is now a latent incident. The Karpenter cost posts on this site, Karpenter vs Cluster Autoscaler and EKS Spot with Karpenter, both lean on flexible instance selection; this error is what happens when that flexibility includes CPU architecture and your images were never told.

A binary was copied in from the wrong place. A multi-stage Dockerfile that COPYs a locally pre-built Go or Rust binary, rather than compiling inside the build, inherits whatever architecture the developer's machine produced. The image config says amd64 because the base image is amd64, but the one file that matters is arm64. Inspect the binary itself:

docker run --rm --platform linux/amd64 --entrypoint sh "$IMAGE" -c 'head -c 20 /app/server | od -An -c'

 177   E   L   F 002 001 001  \0  \0  \0  \0  \0  \0  \0  \0  \0 002  \0 267  \0

Byte 18 of an ELF header is the machine type: \0 076 (0x3e) is x86-64, \0 267 (0xb7) is aarch64. If the image has file installed, file /app/server says it in plain English. Here the binary is aarch64 inside an amd64 image, which is why the image manifest looked fine.

Fix A: Build a multi-arch image and stop caring where it lands

The durable fix is an image whose manifest list covers every architecture your cluster can schedule onto. With BuildKit that is one flag, provided the Dockerfile compiles for the target platform instead of copying a host-built artifact in.

# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# cross-compile on the native build host; no emulation needed for Go
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/server /server
ENTRYPOINT ["/server"]

--platform=$BUILDPLATFORM pins the compiler stage to the runner's own CPU so it runs at full speed, and TARGETARCH tells the Go toolchain what to emit. The final stage is pulled per target platform automatically. The multi-stage builds post covers the layering side of this pattern in more depth.

The GitHub Actions job:

jobs:
  image:
    runs-on: ubuntu-24.04
    permissions: { contents: read, packages: write }
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3        # only needed if a RUN step must execute foreign binaries
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} }
      - uses: docker/build-push-action@v6
        with:
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ghcr.io/${{ github.repository }}/api:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - name: assert both platforms are in the manifest list
        run: |
          IMAGE=ghcr.io/${{ github.repository }}/api:${{ github.sha }}
          have=$(docker buildx imagetools inspect "$IMAGE" --raw \
            | jq -r '[.manifests[] | select(.platform.architecture != "unknown") | .platform.architecture] | sort | join(",")')
          test "$have" = "amd64,arm64" || { echo "manifest has [$have], want amd64,arm64"; exit 1; }

The last step is the part most pipelines skip. A build that silently produced a single-arch image because someone removed the platforms line will pass every other check and fail only in production, on a Tuesday, on the one node that happens to be Graviton. Make the manifest shape a CI assertion, the same way the manifest review agent makes probes and resource limits a CI assertion.

For interpreted languages (Node, Python) the RUN npm ci or pip install step has to execute inside the target platform, which means QEMU emulation on an x86 runner. It works, but expect 3 to 8 times slower builds for the arm64 half. Native arm64 runners (ubuntu-24.04-arm on GitHub) building each half natively and then merging with docker buildx imagetools create are the fast path once the emulated build becomes the long pole of your pipeline.

Fix B: Pin the pod to the architecture it was built for

When you cannot rebuild today, a vendor image is single-arch, or a compiled dependency has no arm64 wheel, constrain scheduling instead. Node affinity on the workload:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values: ["amd64"]

A plain nodeSelector: { kubernetes.io/arch: amd64 } does the same job with less YAML. If the pods then sit in Pending with didn't match Pod's node affinity/selector, the cluster has no spare capacity on that architecture, which is the problem covered in the Pending / FailedScheduling guide.

If Karpenter is provisioning nodes, fix it at the source as well, so the autoscaler cannot create capacity your images cannot run on:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]        # widen to ["amd64", "arm64"] only once every image is multi-arch

The right long-term state for a mixed cluster is Fix A for everything you build, and Fix B as an explicit, documented exception for the handful of third-party images that are stuck on one architecture.

Step 3: It is a script, and the architecture matches

If the node and image architectures agree and the entrypoint is a shell script, the kernel is complaining about the script's first line. execve decides how to run a text file from its first two bytes. If they are not #!, Linux returns ENOEXEC, and you get exactly the same exec format error as a wrong-arch binary.

docker run --rm --entrypoint sh "$IMAGE" -c 'head -c 16 /docker-entrypoint.sh | od -An -c'

 357 273 277   #   !   /   b   i   n   /   s   h  \n   s   e   t

Those first three bytes, 357 273 277 (EF BB BF), are a UTF-8 byte order mark that a Windows editor inserted before the shebang. To the kernel the file now starts with garbage, not #!. The fix is to strip it in the repo and, to stop it recurring, add a .gitattributes and an .editorconfig rule:

sed -i '1s/^\xEF\xBB\xBF//' docker-entrypoint.sh
printf '*.sh text eol=lf\n' >> .gitattributes

A missing shebang entirely (the file starts with set -e or a comment) fails identically. Either add #!/bin/sh as line one or call the interpreter explicitly in the Dockerfile with ENTRYPOINT ["sh", "/docker-entrypoint.sh"], which sidesteps the shebang parse altogether.

One disambiguation that saves time: a shebang with Windows line endings (#!/bin/sh\r) or a shebang pointing at /bin/bash in an Alpine image produces no such file or directory, not exec format error. If you see the former, you are hunting a CRLF or a missing interpreter, not a BOM.

Step 4: The error is in your CI build, not in the cluster

#12 [linux/arm64 build 3/6] RUN apk add --no-cache ca-certificates
#12 0.312 exec /bin/sh: exec format error

This is the build-time version. You asked BuildKit for a linux/arm64 image on an x86 runner, a RUN step needs to execute arm64 /bin/sh, and the runner's kernel has no handler registered to emulate it. The handler is QEMU user-mode registered through binfmt_misc.

# is a handler registered?
ls /proc/sys/fs/binfmt_misc/ | grep qemu-aarch64 || echo "no arm64 handler"

# register handlers for every common target (needs --privileged, once per boot)
docker run --privileged --rm tonistiigi/binfmt --install arm64,amd64

In GitHub Actions docker/setup-qemu-action@v3 does this for you, which is why it sits before setup-buildx-action in the workflow above. On self-hosted runners, Kubernetes-hosted runners in particular, the registration does not survive a node reboot unless you install it on the host or run the binfmt container as a privileged DaemonSet. If arm64 builds succeed on hosted runners and fail on your own, this is the cause almost every time.

The cleaner answer is to not emulate at all: use FROM --platform=$BUILDPLATFORM plus a cross-compiler as in Fix A, so no foreign binary ever executes during the build and QEMU is irrelevant.

A repeatable checklist

1. kubectl describe pod shows Reason: StartError, exit 128, exec format error. Confirm it is this error and not a pull failure.
2. Node arch: kubectl get node $NODE -o jsonpath='{.status.nodeInfo.architecture}'.
3. Image arch: crane manifest for the platform list, crane config if it is single-arch. Ignore unknown/unknown attestation entries.
4. Mismatch found: rebuild multi-arch with buildx and TARGETARCH (Fix A), or pin with kubernetes.io/arch affinity and a Karpenter requirement (Fix B).
5. No mismatch: dump the first 16 bytes of the entrypoint. Fix a BOM or missing shebang. CRLF gives a different error.
6. Error appears in CI instead of the cluster: register QEMU via setup-qemu-action or tonistiigi/binfmt, or cross-compile so nothing foreign executes.
7. Add the manifest-list assertion to the pipeline so a single-arch image can never reach the registry unnoticed again.

The whole sequence takes under five minutes once you have seen it twice. The expensive version is the one where a mixed-architecture node group goes live on a Friday and twenty single-arch services discover it for you over the weekend. Audit the manifests before you add the nodes.