Skip to content
VenerableNetworks
ProfessionalProLoad BalancingNACL / Security GroupsAWS VPC

Lab 06: The Load Balancer That Was Not There

An internal Network Load Balancer with one healthy target. The target group reports healthy for the entire incident, the listener is correct, and the security group permits the load balancer subnet exactly as intended. Every client connection times out.

Debugging time
~30 min
Reading time
12 min
Reported by
Platform Engineering
Tier
Professional
INC-1388SEV-3InvestigatingOpened 2026-09-17 11:40 UTC

Clients time out against the new internal load balancer while its target reports healthy

Reported by Platform Engineering

We put an internal Network Load Balancer in front of the session store so we can add a second node without changing client configuration. The load balancer came up, the target registered, and target health has been healthy continuously since.

No client can connect. Every attempt times out.

The target's security group was written to allow only the load balancer, using the load balancer subnet's CIDR, because we did not want the whole VPC reaching the session store directly. That rule is present and correct. Health checks are clearly getting through, since the target is healthy — so the load balancer can obviously reach it.

What you are working with

One VPC, one Availability Zone, three subnets on distinct ranges.

| Resource | Configuration | | --- | --- | | subnet-lb | 10.40.1.0/24 — the load balancer's network interface | | subnet-app | 10.40.2.0/24 — the target instance | | subnet-client | 10.40.3.0/24 — the calling client | | Load balancer | Internal NLB, TCP 5432, no security group attached | | Target group | Instance type, TCP 5432, TCP health check on the traffic port | | sg-app | Inbound 5432 from 10.40.1.0/24 | | sg-client | All outbound |

  1. EC2

    Client (subnet-client)

    10.40.3.x → nlb:5432

  2. GW

    Network Load Balancer

    Forwards to the target — without rewriting the source address

  3. RTB

    sg-app on the target ENI

    Permits 5432 from 10.40.1.0/24 only

    Dropped — the packet still carries source 10.40.3.x, which the rule does not match — the load balancer never appears as the source

  4. DEST

    Session store (subnet-app)

    10.40.2.x:5432 — listening, and reporting healthy

Health checks originate from the load balancer's own interface in 10.40.1.0/24, so they match the rule and pass. Client traffic does not, because a Network Load Balancer does not put itself in the source field.

Scope and constraints

  • In scope: why client traffic is dropped while health checks succeed.
  • Out of scope: the listener, the target registration, routing, and NACLs. All are correct.
  • Cross-zone load balancing is not the answer. Everything is in one Availability Zone deliberately. AWS removes a zone's address from DNS when it has no healthy target, so an empty zone cannot produce this symptom.
  • The target genuinely is healthy for the whole incident. Treat that as a clue, not as noise.
  • There are three defensible fixes and they have materially different consequences. Finding the cause is the first half of this lab; choosing is the second.

Deploy the broken state

cd lab-06-nlb-client-ip-preservation
terraform init
terraform apply
 
# Shell on the client
aws ssm start-session --target "$(terraform output -raw client_instance_id)"

Give it about two minutes after apply — the listener has to start and the target group needs two successful checks at a ten-second interval before it reports healthy.