Lab 06: The Load Balancer That Was Not There
An internal Network Load Balancer with one healthy target. The target group reports healthy for the entire incident, the listener is correct, and the security group permits the load balancer subnet exactly as intended. Every client connection times out.
- Debugging time
- ~30 min
- Reading time
- 12 min
- Reported by
- Platform Engineering
- Tier
- Professional
Clients time out against the new internal load balancer while its target reports healthy
Reported by Platform Engineering
We put an internal Network Load Balancer in front of the session store so we can add a second node
without changing client configuration. The load balancer came up, the target registered, and target
health has been healthy continuously since.
No client can connect. Every attempt times out.
The target's security group was written to allow only the load balancer, using the load balancer subnet's CIDR, because we did not want the whole VPC reaching the session store directly. That rule is present and correct. Health checks are clearly getting through, since the target is healthy — so the load balancer can obviously reach it.
What you are working with
One VPC, one Availability Zone, three subnets on distinct ranges.
All three subnets share one route table and one Availability Zone, so neither routing nor cross-zone behaviour is in play. The load balancer deliberately has no security group of its own, which leaves exactly one filter on the path: the rule on the target, and it names a single one of these three ranges.
| Resource | Configuration |
|---|---|
subnet-lb | 10.40.1.0/24 — the load balancer's network interface |
subnet-app | 10.40.2.0/24 — the target instance |
subnet-client | 10.40.3.0/24 — the calling client |
| Load balancer | Internal NLB, TCP 5432, no security group attached |
| Target group | Instance type, TCP 5432, TCP health check on the traffic port |
sg-app | Inbound 5432 from 10.40.1.0/24 |
sg-client | All outbound |
- EC2
Client (subnet-client)
10.40.3.x → nlb:5432
- GW
Network Load Balancer
Forwards to the target — without rewriting the source address
- RTB
sg-app on the target ENI
Permits 5432 from 10.40.1.0/24 only
Dropped — the packet still carries source 10.40.3.x, which the rule does not match — the load balancer never appears as the source
- DEST
Session store (subnet-app)
10.40.2.x:5432 — listening, and reporting healthy
Health checks originate from the load balancer's own interface in 10.40.1.0/24, so they match the rule and pass. Client traffic does not, because a Network Load Balancer does not put itself in the source field.
Scope and constraints
- In scope: why client traffic is dropped while health checks succeed.
- Out of scope: the listener, the target registration, routing, and NACLs. All are correct.
- Cross-zone load balancing is not the answer. Everything is in one Availability Zone deliberately. AWS removes a zone's address from DNS when it has no healthy target, so an empty zone cannot produce this symptom.
- The target genuinely is healthy for the whole incident. Treat that as a clue, not as noise.
- There are three defensible fixes and they have materially different consequences. Finding the cause is the first half of this lab; choosing is the second.
Deploy the broken state
cd lab-06-nlb-client-ip-preservation
terraform init
terraform apply
# Shell on the client
aws ssm start-session --target "$(terraform output -raw client_instance_id)"Give it about two minutes after apply — the listener has to start and the target group needs two successful checks at a ten-second interval before it reports healthy.
The rest of this lab is part of Labs Pro
The reproduction steps, the deployable Terraform, and the debrief with root cause and remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The incident brief is open, so you can see what you would be solving before you decide.
Already bought it? Sign in and it unlocks.
The rest of this lab is part of Labs Pro
The reproduction steps, the deployable Terraform, and the debrief with root cause and remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The incident brief is open, so you can see what you would be solving before you decide.
Already bought it? Sign in and it unlocks.