Skip to content
VenerableNetworks
ProfessionalProLoad Balancing

Lab 14: The Health Check That Sent Binary

The target is running and curling it directly returns the right response. The target group says unhealthy, so the load balancer has nothing to route to. The security group permits the load balancer, the port is right, and the health check path exists.

Debugging time
~25 min
Reading time
11 min
Reported by
API Platform
Tier
Professional
INC-1638SEV-2InvestigatingOpened 2026-09-23 15:47 UTC

All targets unhealthy behind the internal NLB after enabling client IP visibility

Reported by API Platform

Every target behind the internal NLB went unhealthy after a change last night. Clients get nothing.

The application is running. I can curl the target's private address on 8080 from another host in the VPC and it returns the expected body. The security group allows 8080 from the whole VPC CIDR, so it covers the load balancer's addresses. The health check path is /, which exists.

The change was enabling proxy protocol on the target group so the application could log real client IPs. That should not affect health checks — health checks come from the load balancer, not from clients.

Target health reason is Target.FailedHealthChecks. That is all we get.

What you are working with

One VPC, one Availability Zone, one internal Network Load Balancer, one target.

Starting state — client, load balancer and target all share one subnet
VPC 10.120.0.0/16 Attached: Internet gateway. AZ a contains lab 10.120.1.0/24, default route 0.0.0.0/0, holding Client host, Internal NLB interface, Target host.the internetVPC 10.120.0.0/16AZ alabpublic10.120.1.0/24Client hostcurl onlyInternal NLB interfaceTCP listener on 80Target hosthttp.server on 80800.0.0.0/0Internet gatewaytooling only

Everything is in one subnet, one zone and one security group scope, so routing, zones and filtering are all off the table before you start. The listener takes TCP on 80 and the target group forwards to 8080, and the target security group admits 8080 from the whole VPC CIDR. The client can reach the target directly on 8080; only the path through the load balancer fails, so whatever differs has to be something the load balancer does to the connection.

ResourceConfiguration
NLBInternal, TCP listener on 80
Target groupTCP, port 8080, target type instance
Target group attributeproxy_protocol_v2.enabled = true
Health checkHTTP on /, port traffic-port, matcher 200
Targetpython3 -m http.server 8080, serving index.html
sg-targetInbound 8080 from 10.120.0.0/16

The health check is HTTP while the listener is TCP. That combination is normal and supported, and it is also the reason this failure is as sharp as it is.

  1. EC2

    NLB node

    10.120.1.x — opens TCP to the target on 8080

  2. FILTER

    sg-target

    inbound 8080 from the VPC CIDR — permitted

  3. VPCE

    Proxy protocol v2 header

    12-byte binary signature, then the header block

  4. DEST

    python http.server

    cannot parse a request line, answers 400

    Dropped — The health check expects 200. A 400 is a valid HTTP response and a failed check, so the target is marked unhealthy while being entirely healthy.

Nothing is dropped at the network layer. The connection succeeds, the exchange completes, and the answer is the wrong number.

Scope and constraints

  • In scope: why the health check fails when a direct request succeeds.
  • Out of scope: the security group, the subnet, the listener, the health check path, and the application. All correct.
  • The target really is healthy. The reporter is right about that.
  • This is not Lab 06. Client IP preservation is a different attribute with a different failure. Here the connection reaches the target and the target answers.

Deploy the broken state

cd lab-14-nlb-proxy-protocol
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw client_instance_id)"

The load balancer takes a few minutes to provision. Health checks begin once it is active.