Skip to content
VenerableNetworks
ProfessionalProLoad Balancing

Lab 14: The Health Check That Sent Binary

The target is running and curling it directly returns the right response. The target group says unhealthy, so the load balancer has nothing to route to. The security group permits the load balancer, the port is right, and the health check path exists.

Debugging time
~25 min
Reading time
10 min
Reported by
API Platform
Tier
Professional
INC-1638SEV-2InvestigatingOpened 2026-09-23 15:47 UTC

All targets unhealthy behind the internal NLB after enabling client IP visibility

Reported by API Platform

Every target behind the internal NLB went unhealthy after a change last night. Clients get nothing.

The application is running. I can curl the target's private address on 8080 from another host in the VPC and it returns the expected body. The security group allows 8080 from the whole VPC CIDR, so it covers the load balancer's addresses. The health check path is /, which exists.

The change was enabling proxy protocol on the target group so the application could log real client IPs. That should not affect health checks — health checks come from the load balancer, not from clients.

Target health reason is Target.FailedHealthChecks. That is all we get.

What you are working with

One VPC, one Availability Zone, one internal Network Load Balancer, one target.

| Resource | Configuration | | --- | --- | | NLB | Internal, TCP listener on 80 | | Target group | TCP, port 8080, target type instance | | Target group attribute | proxy_protocol_v2.enabled = true | | Health check | HTTP on /, port traffic-port, matcher 200 | | Target | python3 -m http.server 8080, serving index.html | | sg-target | Inbound 8080 from 10.120.0.0/16 |

The health check is HTTP while the listener is TCP. That combination is normal and supported, and it is also the reason this failure is as sharp as it is.

  1. EC2

    NLB node

    10.120.1.x — opens TCP to the target on 8080

  2. FILTER

    sg-target

    inbound 8080 from the VPC CIDR — permitted

  3. VPCE

    Proxy protocol v2 header

    12-byte binary signature, then the header block

  4. DEST

    python http.server

    cannot parse a request line, answers 400

    Dropped — The health check expects 200. A 400 is a valid HTTP response and a failed check, so the target is marked unhealthy while being entirely healthy.

Nothing is dropped at the network layer. The connection succeeds, the exchange completes, and the answer is the wrong number.

Scope and constraints

  • In scope: why the health check fails when a direct request succeeds.
  • Out of scope: the security group, the subnet, the listener, the health check path, and the application. All correct.
  • The target really is healthy. The reporter is right about that.
  • This is not Lab 06. Client IP preservation is a different attribute with a different failure. Here the connection reaches the target and the target answers.

Deploy the broken state

cd lab-14-nlb-proxy-protocol
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw client_instance_id)"

The load balancer takes a few minutes to provision. Health checks begin once it is active.