ProfessionalProLoad Balancing

Lab 26: Two Load Balancers, One Empty Zone, One Set Of 503s

An Application Load Balancer and a Network Load Balancer in the same two zones, pointed at the same single healthy target, both with cross-zone off. The Network Load Balancer serves every request. The Application Load Balancer fails about half of them with 503, and the half it fails is decided by DNS before the request is sent.

Debugging time: ~30 min
Reading time: 10 min
Reported by: Application Platform
Tier: Professional

INC-1891SEV-2InvestigatingOpened 2026-10-01 09:20 UTC

ALB returning 503 on roughly half of requests; only target is healthy; NLB on the same instance is fine

Reported by Application Platform

We moved the service behind an Application Load Balancer last night. Since then about half of all requests get a 503 from the load balancer itself, not from us: the application never sees them and our access log has nothing for them. The other half are fine.

The target group has one instance and it is healthy. It has been healthy the whole time. Health check is HTTP on /, matcher 200, and the instance answers 200.

Here is the part I cannot explain. The network team fronts the same instance with a Network Load Balancer for a different consumer, same two subnets, same instance, same port, and it has not failed once. Same target, two load balancers, one of them broken.

What you are working with

One VPC, two Availability Zones, two load balancers, one target.

Starting state — both load balancers span two zones, the target lives in one

Each load balancer has a node in each enabled zone and, with cross-zone off, each node may only send to targets in its own zone. Zone b has none. The two load balancers agree on that fact and disagree about what to do with it: the Application Load Balancer leaves its zone-b address in DNS, the Network Load Balancer withdraws its own. The client in zone b is about to find out which one it resolved.

Resource	Configuration
ALB	Internal, subnets a and b
ALB target group	HTTP:80, one target in zone a, cross-zone `false`
NLB	Internal, subnets a and b, cross-zone off (default)
NLB target group	TCP:80, same target, HTTP health check
App	`10.26.1.50`, zone a, Python HTTP server, healthy in both groups
Control host	zone b, the client

EC2
Control host
resolves the ALB name
RTB
DNS
two addresses, one per zone
GW
ALB node, zone b
cross-zone off; permitted targets: those in zone b
Dropped — The node is in DNS and accepts the connection, then looks for a target in its own zone. There is none. The load balancer answers 503 itself. The target never sees the request.
DEST
App, zone a
healthy, idle

The request is refused one hop after DNS, by a node that was published as reachable. Nothing on the target side participates.

Scope and constraints

In scope: why two load balancers with identical topology and an identical healthy target produce different results, and what DNS has to do with it.
Out of scope: the application, the health check, security groups, subnets, the Network Load Balancer. All correct.
The reporter is right that the target is healthy and right that the NLB is fine. Both facts are evidence, not noise.
The fix is one target group attribute, or one more instance. The lab does the attribute.

Deploy the broken state

cd lab-26-alb-empty-zone-503
terraform init
terraform apply

About four minutes for both load balancers to become active and the target to pass its initial health checks. Confirm before testing:

aws elbv2 describe-target-health --target-group-arn "$(terraform output -raw alb_target_group_arn)" \
  --query 'TargetHealthDescriptions[].TargetHealth.State' --output text
# healthy
aws elbv2 describe-target-health --target-group-arn "$(terraform output -raw nlb_target_group_arn)" \
  --query 'TargetHealthDescriptions[].TargetHealth.State' --output text
# healthy

ALB=$(terraform output -raw alb_dns_name)
NLB=$(terraform output -raw nlb_dns_name)
terraform output -raw start_session_command

Reproduce the half

On the control host, twenty requests to the Application Load Balancer:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{http_code}\n' "http://$ALB/"
done | sort | uniq -c
#   9 200
#  11 503

And twenty to the Network Load Balancer:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{http_code}\n' "http://$NLB/"
done | sort | uniq -c
#  20 200

Same instance. One load balancer loses roughly half.

Look at DNS before you look at anything else

dig +short "$ALB"
# 10.26.1.213
# 10.26.2.87
 
dig +short "$NLB"
# 10.26.1.144

The Application Load Balancer publishes two addresses. The Network Load Balancer publishes one. That is the whole difference, and the rest of this lab is working out why.

The ALB's addresses are one per zone. Confirm which is which from your workstation, using the load balancer's network interfaces:

aws ec2 describe-network-interfaces \
  --filters "Name=description,Values=ELB app/vn-lab-26-alb/*" \
  --query 'NetworkInterfaces[].[PrivateIpAddress,AvailabilityZone]' --output text
# 10.26.1.213	us-east-1a
# 10.26.2.87	us-east-1b

Make the half deterministic

Back on the control host, address each ALB node directly instead of letting DNS choose:

curl -s -o /dev/null -w '%{http_code}\n' http://10.26.1.213/   # zone a node
# 200
curl -s -o /dev/null -w '%{http_code}\n' http://10.26.2.87/    # zone b node
# 503

Repeat either as many times as you like; the answer does not change. The zone-a node always succeeds. The zone-b node always fails. "Half" was never random: it was DNS handing out two addresses, one of which has nothing behind it.

Confirm what the zone-b node is allowed to do

aws elbv2 describe-target-group-attributes \
  --target-group-arn "$(terraform output -raw alb_target_group_arn)" \
  --query 'Attributes[?Key==`load_balancing.cross_zone.enabled`].Value' --output text
# false

Cross-zone is off on the target group. Each node may only forward to targets in its own zone. The zone-b node's set of permitted targets is empty, and the ALB's documented behaviour for that state is to answer 503.

ALB returning 503 on roughly half of requests; only target is healthy; NLB on the same instance is fine

What you are working with#

Scope and constraints#

Deploy the broken state#

Reproduce the half#

Look at DNS before you look at anything else#

Make the half deterministic#

Confirm what the zone-b node is allowed to do#

This debrief is part of Labs Pro