Kubernetes Ingress 502 Bad Gateway (ingress-nginx): How to Debug and Fix 502, 503, and 504 Errors

What a 502 from the Ingress actually means

A 502 Bad Gateway page that says nginx at the bottom means the ingress-nginx controller accepted the request, tried to open a connection to one of your pods, and did not get a valid HTTP response back. Your Service, your Ingress rule, and TLS all worked. The failure is between the controller and the pod IP: nothing listening on that port, the app closing the connection mid-request, a keepalive race, or a backend that speaks HTTPS or gRPC while nginx sent plain HTTP/1.1. Its siblings are 503, which almost always means the Service has no ready endpoints at all, and 504, which means the pod accepted the connection but took longer than proxy-read-timeout (60 seconds by default) to answer.

The controller writes one access-log line per request that names the exact upstream pod IP it tried, and one error-log line that says why the attempt failed. Between those two lines you can diagnose almost every case in under five minutes. This post is the diagnosis order, the log line for each cause, and the fix.

Step 1: Confirm which hop returned the error

Three different things can print a 502: a cloud load balancer in front of the controller, the controller itself, or your app if it is a proxy too. The access log settles it. ingress-nginx's default log_format includes the upstream name, the pod address it chose, and the status that pod returned:

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --since=10m \
  | grep -E ' (502|503|504) ' | tail -20

10.0.1.5 - - [26/Sep/2026:09:14:02 +0000] "GET /api/orders HTTP/1.1" 502 150 "-" "curl/8.5.0"
  88 0.003 [prod-api-8080] [] 10.0.3.14:8080 0 0.003 502 6f1c9a0e2b3d

Read the tail of the line. [prod-api-8080] is the upstream name, built as namespace-service-port, so you can see at a glance which backend the Ingress rule resolved to. 10.0.3.14:8080 is the pod IP and port nginx actually connected to. The last 502 is the status of that upstream attempt. Three shapes tell you where to look:

If the controller log shows nothing at all for the failing request, the error came from the load balancer in front of it (an idle-timeout or health-check failure on an ALB or NLB) and no amount of Ingress annotation will fix it. Check the target group health first.

The error log explains the why for every 502 in the first shape. Pull it separately, because it is interleaved with access lines:

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --since=10m \
  | grep -E 'connect\(\) failed|prematurely closed|timed out|too big header|no live upstreams|reset by peer'

The rest of this post is organised by what that grep returns.

Step 2: 503 and empty upstreams: no endpoints

ingress-nginx does not route through the Service's ClusterIP. It watches EndpointSlices and load-balances directly to pod IPs. So when a Service has zero ready endpoints, the controller has nowhere to send the request and returns 503 itself. It also logs a warning once per sync:

W0926 09:12:41.117 controller.go:1213] Service "prod/api" does not have any active Endpoint.

Check the endpoints and compare the selector to the pods:

kubectl get endpointslices -n prod -l kubernetes.io/service-name=api
kubectl get svc api -n prod -o jsonpath='{.spec.selector}'; echo
kubectl get pods -n prod -l app=api -o wide

If the EndpointSlice has no addresses but the pods are Running, one of three things is wrong:

• Selector mismatch. The Service selects app: api and the Deployment labels its pods app: api-server. The pods exist, the Service is empty, and nothing errors. Fix the label in one place.
• Readiness probe failing. Pods that are Running but 0/1 ready are deliberately excluded from endpoints. kubectl describe pod shows Readiness probe failed. This is the right behaviour, and the fix is the probe or the app, as covered in the liveness, readiness, and startup probes guide.
• All pods restarting. A deploy pushed a bad image, every replica is in CrashLoopBackOff, and the endpoints emptied out as each one failed readiness. Roll back first, diagnose second.

There is one more 503 that is not about endpoints. If kubectl get ingress shows the right host but kubectl describe ingress has an event like Ingress does not contain a valid IngressClass, the controller has ignored the object entirely and the request is hitting the default backend. Set spec.ingressClassName: nginx explicitly.

To see exactly what the controller believes about a backend, the ingress-nginx kubectl plugin (installed via krew) dumps its live Lua backend table:

kubectl ingress-nginx backends -n ingress-nginx --deployment ingress-nginx-controller \
  | jq -r '.[] | "\(.name)\t\(.endpoints | length) endpoints"'

prod-api-8080        0 endpoints
prod-web-3000        3 endpoints
upstream-default-backend   1 endpoints

Zero here with running pods is always a selector, readiness, or port-name problem, never a network one.

Step 3: connect() failed (111: Connection refused): wrong port or wrong bind address

2026/09/26 09:14:02 [error] 31#31: *8812 connect() failed (111: Connection refused)
  while connecting to upstream, client: 10.0.1.5, server: api.example.com,
  request: "GET /api/orders HTTP/1.1", upstream: "http://10.0.3.14:8080/api/orders"

nginx reached the pod's IP and the kernel on that node answered that nothing is listening on 8080. Two causes account for nearly all of these.

The port chain is broken. Three numbers have to line up: the Ingress backend.service.port.number must match the Service port, the Service targetPort must match what the container binds, and if targetPort is a name it must match a containerPort name in the pod spec. The classic miss is a Service that exposes port: 80 with targetPort: 80 while the app listens on 8080:

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: prod
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 80          # what the Ingress references
      targetPort: 8080  # what the container actually binds

Note that ingress-nginx bypasses the Service for routing but still reads it for the port mapping. The pod IP in the upstream field is right, and the port after the colon is what the controller derived from targetPort. If that port is not the one your app binds, this is your bug.

The app binds to localhost. Frameworks default to 127.0.0.1 more often than people expect: Flask's app.run(), Rails' bin/rails s on older versions, Vite and Next.js dev servers, and anything configured with HOST=localhost. Inside a container, 127.0.0.1 is reachable only from the container's own network namespace. Traffic from the controller arrives on the pod IP and is refused. Confirm it from inside the pod:

kubectl exec -n prod deploy/api -- sh -c 'cat /proc/net/tcp | awk "NR>1 {print \$2}"'

0100007F:1F90 is 127.0.0.1:8080 and means bound to loopback. 00000000:1F90 is 0.0.0.0:8080 and means bound to all interfaces. Fix the bind address to 0.0.0.0 (or :: for dual-stack) in the app's config, not in Kubernetes.

Then test the exact path nginx takes, from the controller pod to the pod IP, so you are not fooled by a Service or DNS layer that is fine:

kubectl exec -n ingress-nginx deploy/ingress-nginx-controller -- \
  curl -s -o /dev/null -w '%{http_code}\n' --max-time 3 http://10.0.3.14:8080/healthz

If this returns 200 while users still get 502s, the controller's backend table is stale; check for a stuck sync in the controller log or restart the controller.

Step 4: connect() failed (110: Operation timed out): NetworkPolicy or security group

Refused means the node said no. Timed out means the packet never arrived or the reply never came back, which points at a NetworkPolicy, a CNI problem, or a cloud security group between nodes. The most common trigger is a team adding a default-deny policy to a namespace and forgetting that the ingress controller lives in a different one. The allow rule needs a namespaceSelector on the controller's namespace label:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress-nginx
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - port: 8080

On clusters running Cilium, hubble observe --to-pod prod/api --verdict DROPPED shows the drop and the policy that caused it in one line. If you are generating policies at scale, the Hubble-flows network policy agent exists precisely to avoid this class of self-inflicted outage.

Step 5: upstream prematurely closed connection: the keepalive race

This is the 502 that only shows up under load, at a rate of a fraction of a percent, and disappears when you look at any individual pod. The error line is:

upstream prematurely closed connection while reading response header from upstream

or its cousin recv() failed (104: Connection reset by peer). It means nginx reused an idle keepalive connection to the pod at the same moment the app decided that connection had been idle too long and closed it. The request went into a socket that was already dead on the other side.

ingress-nginx keeps upstream connections open for upstream-keepalive-timeout seconds, which defaults to 60. Most application servers close idle connections much sooner:

Any app whose idle timeout is shorter than nginx's has this race. The rule is: the app's keepalive timeout must be longer than the proxy's. Fix it from either side. In the controller's ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  upstream-keepalive-timeout: "15"
  upstream-keepalive-connections: "320"
  upstream-keepalive-requests: "10000"

Or in the app, which is better because it fixes the same race for every proxy in front of it. For Node:

const server = app.listen(8080, "0.0.0.0");
server.keepAliveTimeout = 65_000;   // longer than any proxy in front
server.headersTimeout = 66_000;     // must exceed keepAliveTimeout

For gunicorn, --keep-alive 75. For uvicorn, --timeout-keep-alive 75. The same numbers protect you from an AWS ALB, which holds idle connections for 60 seconds and produces the identical intermittent 502 for the identical reason.

The other source of prematurely closed is not a race at all: the pod died mid-request. kubectl get pods shows a restart count climbing, and the last state is usually OOMKilled with exit code 137. Every in-flight request on that pod becomes a 502 at the moment the kernel kills it.

Step 6: upstream timed out (110): the 504 case

upstream timed out (110: Operation timed out) while reading response header from upstream

The connection opened, the request was sent, and the pod did not return response headers within proxy-read-timeout. The default is 60 seconds, which is generous for an API and far too short for a report export or a long-polling endpoint. Raise it per Ingress, not globally:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"

Before you raise it, check that the load balancer in front allows the new number. An ALB's idle timeout defaults to 60 seconds and an NLB's to 350. Setting proxy-read-timeout to 300 behind a default ALB moves the 504 from nginx to the ALB and changes nothing for the user. And if a request routinely takes more than a minute, the real fix is usually an async job with a polling endpoint, not a longer timeout.

Step 7: Backend speaks HTTPS or gRPC

If the app terminates its own TLS, or serves gRPC, nginx's plain HTTP/1.1 request is garbage to it. The symptoms vary: the app logs http: TLS handshake error, the controller logs upstream sent no valid HTTP/1.0 header, or an nginx-based app returns 400 The plain HTTP request was sent to HTTPS port which the controller passes through. Tell the controller what protocol to use:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"   # or "GRPC", "GRPCS"

For gRPC the Ingress host must also have TLS configured, because gRPC over the public side of ingress-nginx requires HTTP/2, and ingress-nginx only negotiates HTTP/2 on TLS listeners. A gRPC backend with an http-only Ingress fails with a 502 that no backend change will fix.

Step 8: upstream sent too big header: response headers over the buffer

upstream sent too big header while reading response header from upstream

The pod's response headers exceed nginx's proxy_buffer_size, which defaults to 4k in ingress-nginx. The usual culprits are large session cookies from OAuth or Spring Security and verbose Set-Cookie bursts after login. Raise the buffer for that Ingress:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"

Or globally with proxy-buffer-size: "16k" in the controller ConfigMap. This is the one 502 that will reproduce on exactly one endpoint, right after authentication, and nowhere else.

The rolling-deploy blip

A short burst of 502s and 503s on every deploy, then nothing, is a specific and fixable thing. When a pod is terminated, two things happen concurrently: the kubelet sends SIGTERM to the container, and the endpoint controller removes the pod IP from the EndpointSlice. ingress-nginx then has to observe that change and update its backend table. If the app exits the instant it receives SIGTERM, there is a window of a second or two where nginx still routes to a pod that is gone. Requests in that window get connection refused or prematurely closed.

The fix is to make the old pod keep serving while the controller catches up. The sleep lifecycle action has been enabled by default since Kubernetes 1.30 and GA since 1.32, and it needs no shell in the image:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      lifecycle:
        preStop:
          sleep:
            seconds: 5

Pair it with an app that handles SIGTERM by finishing in-flight requests before exiting, and the blip disappears. The zero-downtime GitHub Actions deployment post walks through the full rollout configuration this belongs to, including maxUnavailable: 0.

Quick reference

Most of these are ordinary nginx behaviour with a Kubernetes-shaped cause. If you tune ingress-nginx often, the nginx reverse proxy tuning guide covers the buffer, keepalive, and timeout directives the annotations map onto, and knowing those makes the controller's ConfigMap far less mysterious.

Related Reading

• Kubernetes Liveness, Readiness, and Startup Probes: A Practical Guide — readiness is what decides whether a pod is in the endpoint list the controller routes to.
• Kubernetes CrashLoopBackOff: How to Debug and Fix It — when a 503 is really every replica of a new image failing.
• Kubernetes DNS Resolution Failures: How to Debug and Fix CoreDNS Issues — the failure that produces a 502 from an app that is itself a proxy and cannot resolve its own upstream.
• Kubernetes OOMKilled (Exit Code 137): How to Debug and Fix It — the usual reason a pod disappears mid-request.