How to Fix AWS ALB Unhealthy Targets and 502/503 Errors (2026)

This guide is for anyone looking at an AWS Application Load Balancer target group full of red "unhealthy" targets, or a dashboard of HTTPCode_ELB_502_Count spikes, and wanting the cause rather than a guess. It works cause by cause, starting from the reason code AWS already gives you, and covers plain EC2 and ECS targets as well as EKS with the AWS Load Balancer Controller. By the end you will know which component is lying (health check, network, Kubernetes rollout or application timeout) and have the exact attribute or annotation to change. Commands use AWS CLI v2 elbv2 and AWS Load Balancer Controller v3.6.2 (Helm chart 3.6.2), the current release as of November 2026. You need read access to the load balancer and, for the EKS steps, kubectl access to the cluster. If you want the background on L4 vs L7, timeouts and LCU pricing first, read the companion explainer, Load Balancers Explained (2026).

Step 1: Read the target health reason code

Do not start by changing settings. AWS records why each target is unhealthy, and the reason code splits the problem space in half: codes starting with Elb come from the load balancer side and codes starting with Target from the target side, according to the health check documentation.

# Find the target group ARN by name, then ask why each target is unhealthy.
TG_ARN=$(aws elbv2 describe-target-groups --names my-app-tg \
  --query 'TargetGroups[0].TargetGroupArn' --output text)

aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]' \
  --output table

The output looks roughly like this when the health check path is wrong (illustrative values):

-----------------------------------------------------------------------------------------------
|                                    DescribeTargetHealth                                     |
+--------------+-------+------------+-------------------------------+-------------------------+
|  10.0.12.41  |  8080 |  unhealthy |  Target.ResponseCodeMismatch  |  Health checks failed with these codes: [404] |
|  10.0.13.77  |  8080 |  unhealthy |  Target.ResponseCodeMismatch  |  Health checks failed with these codes: [404] |
+--------------+-------+------------+-------------------------------+-------------------------+

Map the reason to a step in this guide:

Reason code	Meaning	Go to
Target.ResponseCodeMismatch	The app answered, with the wrong status	Step 2
Target.Timeout	No answer within the health check timeout	Step 3
Target.FailedHealthChecks	Connection failed, or app not listening on the port	Steps 3 and 4
Elb.InitialHealthChecking	Still running the first checks	Wait; a new target needs one passing check
Target.NotInUse	Target group not used by a listener rule, or AZ not enabled	Step 7
Target.DeregistrationInProgress	Target is draining	Step 5

Also print the health check settings, since you will need them in every step:

aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
  --query 'TargetGroups[0].{Path:HealthCheckPath,Port:HealthCheckPort,Proto:HealthCheckProtocol,Interval:HealthCheckIntervalSeconds,Timeout:HealthCheckTimeoutSeconds,Healthy:HealthyThresholdCount,Unhealthy:UnhealthyThresholdCount,Codes:Matcher.HttpCode,Type:TargetType}'

Step 2: Fix the health check path, success codes and Host header

Target.ResponseCodeMismatch means the target answered, just not with a code in the matcher. The default matcher is 200, so three things cause most of these.

The path redirects. A health check on / hits an app that sends 301 to HTTPS or 302 to /login. The ALB does not follow redirects. Point the check at a dedicated endpoint that returns 200 with no auth.

The app rejects the Host header. The ALB sends health checks with the target's private IP as the Host, for example Host: 10.0.0.10:8080, and a User-Agent of ELB-HealthChecker/2.0, per the ALB troubleshooting page. Frameworks with an allowed-hosts list (Django's ALLOWED_HOSTS, Rails host authorization) or nginx with only named server blocks answer 400 or 404. Allow the health path through, or add a default server.

The endpoint really fails. If /healthz checks the database, a slow database fails every target at once. Keep it shallow.

Test from inside the VPC, with the same Host header the ALB will use:

# Run from a bastion or a pod in the same VPC. Expect "200".
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'Host: 10.0.12.41:8080' -A 'ELB-HealthChecker/2.0' \
  http://10.0.12.41:8080/healthz

Then fix the target group. Success codes accept a list or range from 200 to 499:

aws elbv2 modify-target-group --target-group-arn "$TG_ARN" \
  --health-check-path /healthz \
  --matcher HttpCode=200-399 \
  --health-check-interval-seconds 15 \
  --healthy-threshold-count 2

On EKS, change the Ingress annotations instead, or the controller will put the old values back on its next reconcile:

# Ingress annotations read by AWS Load Balancer Controller v3.6.2
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/success-codes: "200-399"
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"

Step 3: Fix security groups and network ACLs

Target.Timeout and Target.FailedHealthChecks usually mean packets never arrived. The target's security group must allow the health check port from the load balancer's security group, and the ALB's security group must allow outbound to the targets on both the traffic and health check ports. Network ACLs are stateless, so the target subnets must also allow outbound ephemeral ports 1024 to 65535 back to the ALB subnets. Get the ALB's security groups and look at the target's inbound rules:

LB_ARN=$(aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
  --query 'TargetGroups[0].LoadBalancerArns[0]' --output text)
aws elbv2 describe-load-balancers --load-balancer-arns "$LB_ARN" \
  --query 'LoadBalancers[0].SecurityGroups'

# Allow the ALB security group to reach the app port on the target security group.
aws ec2 authorize-security-group-ingress --group-id sg-0target1234567890 \
  --protocol tcp --port 8080 --source-group sg-0alb12345678901234

Also confirm the app listens on the right interface. A process bound to 127.0.0.1:8080 passes a curl localhost test on the box and fails every health check. ss -ltnp | grep 8080 should show 0.0.0.0:8080 or the instance IP.

On EKS, the controller manages these rules for you when it creates the security groups. If you pass your own with alb.ingress.kubernetes.io/security-groups, set alb.ingress.kubernetes.io/manage-backend-security-group-rules: "true" or add the node or pod security group rules yourself. For the VPC side of this, see AWS VPC Explained.

Step 4: Choose ip vs instance target type on EKS

The controller annotation reference defaults alb.ingress.kubernetes.io/target-type to instance. In instance mode the ALB registers nodes and sends traffic to the Service's NodePort, so the Service must be NodePort or LoadBalancer, and every node is health-checked through kube-proxy. A node with no ready pod for that Service can still pass or fail depending on externalTrafficPolicy, which is confusing to debug.

ip mode registers pod IPs directly, works with a ClusterIP Service, and needs a CNI that gives pods VPC IPs, such as the Amazon VPC CNI. It is also required for readiness gates in the next step. For most clusters I would switch:

metadata:
  annotations:
    alb.ingress.kubernetes.io/target-type: ip

After switching, describe-target-health should list pod IPs on the container port (8080), not node IDs on a 30000-range port. If targets show Target.FailedHealthChecks on the container port, check the pod security group or the node security group rules for that port.

Step 5: Stop 502s and 503s during deploys with readiness gates and draining

If targets go unhealthy only during rollouts, the problem is timing, not configuration. Two races happen.

On the way in, Kubernetes marks a new pod Ready when its readiness probe passes and the Deployment kills an old pod, even though the ALB has not finished its own health checks on the new one. Fix it with the controller's pod readiness gate. It works only with target-type: ip, and only for pods created after the namespace is labelled:

# Install or upgrade the controller (chart 3.6.2 ships app v3.6.2).
# helm upgrade does not apply CRDs; apply them first as the chart README says.
helm repo add eks https://aws.github.io/eks-charts
helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --version 3.6.2 -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

# Turn on readiness gate injection, then recreate the pods.
kubectl label namespace shop elbv2.k8s.aws/pod-readiness-gate-inject=enabled
kubectl -n shop rollout restart deployment/api

# READINESS GATES shows 0/1 until the ALB reports the pod healthy, then 1/1.
kubectl -n shop get pods -o wide

The docs note the webhook's failurePolicy defaults to Ignore after v2.11.0, so if the webhook is down pods start without the gate. Check that new pods actually carry a target-health.elbv2.k8s.aws/... condition.

On the way out, the ALB keeps sending requests to a terminating pod until deregistration finishes, and the default deregistration delay is 300 seconds. AWS lists "the deregistration delay period elapsed for a request being handled by a target that was deregistered" as a 502 cause, and the pod exiting before deregistration completes gives the same result. Give the pod a short preStop pause so the ALB stops routing first, keep terminationGracePeriodSeconds above that, and shorten the delay if your requests are quick:

# Deployment pod spec excerpt
spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      image: registry.example.com/api:1.8.2
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "20"]   # image must contain a sleep binary
---
# Ingress annotation: drain for 30 s instead of 300 s
metadata:
  annotations:
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30

Step 6: Fix intermittent 502s caused by keep-alive timeouts

If every target is healthy and you still see a trickle of 502s, often with near-zero target_processing_time, look at keep-alive. The ALB reuses back-end connections and its idle timeout (idle_timeout.timeout_seconds) defaults to 60 seconds. If the target closes an idle connection sooner, the ALB can send a request just as the target sends FIN, and the client gets a 502. AWS says to check "whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer."

Node.js is the usual suspect: server.keepAliveTimeout defaults to 5,000 ms. Set it above the ALB idle timeout and keep headersTimeout above that:

// server.js (Node.js 22 or later). ALB idle timeout is 60 s, so outlast it.
const http = require("node:http");
const server = http.createServer(app);
server.keepAliveTimeout = 65_000;  // ms, must exceed the ALB idle timeout
server.headersTimeout = 66_000;    // ms, keep above keepAliveTimeout
server.listen(8080);

nginx defaults to keepalive_timeout 75s, which is already safe behind a 60-second ALB. If you raised the ALB idle timeout, for example to 300 for long requests, raise every backend's keep-alive to match. Check the current value:

aws elbv2 describe-load-balancer-attributes --load-balancer-arn "$LB_ARN" \
  --query "Attributes[?Key=='idle_timeout.timeout_seconds' || Key=='client_keep_alive.seconds']"

Step 7: Fix 503 and 504 errors

A 503 from the ALB has a short list of causes in the troubleshooting guide: the target group has no registered targets, or all of them are unused. unused covers a target group not attached to any listener rule and targets in an Availability Zone the ALB does not have enabled (Target.NotInUse). On EKS, an empty target group almost always means the Service selector matches no pods, or the controller failed to reconcile. Check both:

kubectl -n shop get endpointslices -l kubernetes.io/service-name=api
kubectl -n shop describe ingress api | sed -n '/Events/,$p'
kubectl -n kube-system logs deploy/aws-load-balancer-controller --since=15m | grep -i error

With cross-zone load balancing turned off at the target group level, an AZ with no healthy targets returns 503 for requests that land on that zone's node. The target group attribute docs also warn against empty subnets in the ALB for the same reason.

A 504 means the ALB connected but the target did not answer before the idle timeout, or could not connect within 10 seconds. Either the request is slow (raise idle_timeout.timeout_seconds or make it async) or NACLs are blocking return traffic on ephemeral ports. Remember that if every target fails health checks the ALB fails open and routes to all of them, so "all unhealthy" plus 504s usually means a network block rather than dead apps. For the same errors from nginx and other proxies, see How to Fix 502 Bad Gateway and 504 Gateway Timeout Errors.

Step 8: Turn on access logs and separate ALB errors from app errors

CloudWatch splits HTTPCode_ELB_5XX_Count (the ALB generated it) from HTTPCode_Target_5XX_Count (your app returned it). Access logs tell you which request did what. Enable them on the Ingress:

metadata:
  annotations:
    alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=my-alb-logs,access_logs.s3.prefix=shop

The ALB writes a file per node every five minutes. In the access log format, field 7 is target_processing_time, field 9 is elb_status_code and field 10 is target_status_code. An ALB-generated 502 has target_status_code set to -, and -1 in the timing fields means the request was never answered:

# Count ALB-generated 5xx by target IP:port over downloaded log files.
zcat *.log.gz | awk '$9 ~ /^5/ && $10 == "-" {print $9, $5}' | sort | uniq -c | sort -rn | head

A 502 concentrated on one target IP points at that pod or instance. Spread across all targets with -1 timings points at keep-alive (Step 6) or deploy timing (Step 5).

Troubleshooting

Symptom	Likely cause	Fix
Target.ResponseCodeMismatch with [301] or [302]	Health check path redirects to HTTPS or login	Use a dedicated /healthz, or widen matcher to 200-399
Target.ResponseCodeMismatch with [400] or [404]	App rejects Host: <private-ip>	Allow the health path for any host, or add a default server block
Target.Timeout on every target	Security group or NACL blocks the health check port	Allow the ALB security group on the port; open ephemeral return ports in NACLs
Target.FailedHealthChecks, works with curl localhost	App bound to 127.0.0.1 or wrong port	Bind to 0.0.0.0; check healthcheck-port matches the container port
Targets are node IDs on 30000-range ports	target-type is instance (the default)	Set alb.ingress.kubernetes.io/target-type: ip
502/503 bursts only during rollouts	No readiness gate; pods exit before deregistration	Label namespace for readiness gates, add preStop sleep, tune deregistration delay
Random 502s, all targets healthy	Backend keep-alive shorter than 60 s idle timeout	Raise app keep-alive above the ALB idle timeout (Node.js: 65 s)
503 with an empty target group	Service selector matches no pods, or controller error	Check EndpointSlices and controller logs
504 on long requests	Request exceeds idle timeout	Raise idle_timeout.timeout_seconds or make the work async
Settings revert after a few minutes	Edited in console, controller reconciled	Change Ingress or Service annotations instead

FAQ

Why are my ALB targets unhealthy when the app works?

Usually because the health check is not making the request you think. It goes to the target's private IP with that IP as the Host header, on the health check port, and expects exactly the success codes in the matcher (200 by default). Reproduce it with curl from inside the VPC using the same Host header and you will normally see the 301, 404 or timeout yourself.

What is the difference between an ALB 502 and 503?

A 502 means the ALB reached a target but the exchange failed: a reset, a malformed response, a closed keep-alive connection or a TLS error. A 503 means there was nothing to send the request to, because the target group had no registered or usable targets. A 504 means the target was too slow.

Do I need pod readiness gates with the AWS Load Balancer Controller?

If you use target-type: ip and do rolling deployments, yes. Without them Kubernetes can terminate old pods before the ALB has marked the new ones healthy, which produces 502 and 503 bursts on every deploy.

Should I lower the deregistration delay?

Lower it to slightly above your longest normal request time, for example 30 seconds for an API, so deploys finish faster. Do not set it below your slowest request, or the ALB will cut those requests off with 502s.

When an ALB goes red in production I check things in this order: the reason code from describe-target-health, then a curl from inside the VPC with the ALB's Host header, then whether the ELB or Target 5xx metric is the one moving, and only then timeouts. Most cases end at step two. The rest are nearly always keep-alive or rollout timing, both easy to fix and annoying to find without access logs, so I turn those on before anything breaks. The official references are the ALB troubleshooting guide, the target group health check reference and the AWS Load Balancer Controller docs.

Related: How to Fix 502 Bad Gateway and 504 Gateway Timeout Errors (2026), How to Fix Kubernetes Readiness and Liveness Probe Failed Errors (2026) and How to Upgrade an EKS Cluster to 1.37 Safely: Step by Step (2026).