When the AWS docs contradict themselves: NLB client IP preservation

Most documentation problems are omissions. You look for a behaviour, it is not described, you test it. Unpleasant but tractable.

The harder case is when the documentation is confident and internally inconsistent, because there is nothing to prompt you to test anything. You read one page, get a clear answer, and build on it. This came up while building a Network Load Balancer lab, and the two pages involved are both ones you would reasonably consult.

The two statements#

From Edit target group attributes for your Network Load Balancer, under default settings, instance type target groups are listed as Enabled. The surrounding prose says preservation is enabled by default and cannot be disabled for instance and IP type target groups using UDP, TCP_UDP, QUIC, and TCP_QUIC, and that TCP and TLS target groups are configurable.

From Register targets with your target group, under how client IP preservation works:

Network Load Balancers don't preserve client IP addresses unless you set the preserve_client_ip.enabled attribute to true.

For an instance target group carrying TCP — an extremely ordinary configuration — the first page says preservation is on and the second says it is off. Not ambiguous, not a matter of emphasis. Opposite.

Why this particular disagreement is expensive#

If the two pages disagreed about something cosmetic it would be a curiosity. This one determines the security group rules on your targets, and the same page that contains the incorrect generalization is the page that tells you how to write them.

The rule you need depends on the answer:

• Preservation on: packets arrive at the target with the client's source address. The target's security group has to permit the client IP ranges.
• Preservation off: packets arrive with the load balancer's private address. The target's security group has to permit the load balancer's private IPs.

Those are different rules, and one does not accidentally satisfy the other.

The health-check path is worth stating carefully, because it is also conditional. The register-targets page splits its recommendations by whether the load balancer has an associated security group. If it does, you reference the load balancer's security group for both client and health-check traffic. If it does not, you permit the load balancer's private IP addresses for health checks, and for client traffic you permit either the approved client IPs or the load balancer private IPs depending on preservation. The security group on the load balancer is a third variable most people do not realize is in play.

Which page to trust, and why#

The edit-target-group-attributes page is the one to follow, and the reason generalizes past this example.

That page enumerates by target type and protocol. It distinguishes instance target groups from IP target groups, and within IP target groups it distinguishes UDP, TCP_UDP, QUIC, and TCP_QUIC from TCP and TLS. It produces a different answer for each combination.

The register-targets page states one flat rule. That rule happens to be correct for IP target groups using TCP or TLS, where the default genuinely is disabled. It is an over-generalization of a true statement — someone described the configurable case and dropped the qualifier.

That is the useful heuristic. When two AWS pages disagree, prefer the one that enumerates cases over the one that states a rule. Enumeration is the harder thing to write and the harder thing to get wrong by omission. A flat rule is what you get when a special case is generalized past its boundary.

The resolution: stop reading prose about defaults#

The general lesson is not "read more carefully." Both statements are clear, and reading either one carefully gets you a confident wrong answer half the time.

Defaults are queryable. Ask the API:

aws elbv2 describe-target-group-attributes \
  --target-group-arn "$TG_ARN" \
  --query "Attributes[?Key=='preserve_client_ip.enabled']"

Then set it explicitly in your infrastructure code rather than inheriting it, so the value is visible in review and cannot change underneath you. An attribute you have written down is one you cannot be wrong about, and it makes the security group rules next to it self-documenting.

Related edges on the same attribute#

While confirming the above, several other constraints on this attribute turned out to be documented but easy to miss. All are from the two pages already linked.

PrivateLink inbound traffic is never preserved. The source is always the load balancer's private IP, regardless of the attribute. If you set preservation on and write client-IP security group rules, PrivateLink consumers break while direct clients work.

IPv6-to-IPv4 translated traffic is never preserved either, and preservation only works when client and target are both IPv4 or both IPv6.

Preservation requires a direct path. The target must be in the same VPC or a peered VPC in the same Region. It is not supported through a Transit Gateway, and not supported when a Gateway Load Balancer endpoint sits in the path for inspection — even inside the same VPC.

NAT loopback is unsupported when preservation is on. If a target behind an internal NLB opens a connection to that same NLB, the connection can be routed back to the originating target. The documented options are to not do that, or to disable preservation and recover the client address with Proxy Protocol v2.

Turning preservation off imposes a connection limit. With preservation disabled you get roughly 55,000 simultaneous connections to each unique target, defined as IP address and port, before port allocation errors become likely. That is the same per-destination shape as the NAT Gateway limit in the previous post — the resource being exhausted is source ports toward one endpoint, not connections in aggregate.

That last one matters because "just disable preservation" is the usual advice for every problem above. It is often the right call, but it is a trade, and the thing you trade for is a ceiling that scales per target rather than in total.

Seeing it fail#

Reading that a security group rule was written for the wrong source address is not the same as watching a healthy target refuse traffic. The NLB client IP preservation lab deploys exactly that state: a target group whose health checks pass, a security group rule that is entirely reasonable for a load balancer behaving the other way, and clients that time out.

The lab works through three separate fixes — permit the clients, turn preservation off, or give the load balancer its own security group and reference it — because which one is correct depends on whether you need the client address downstream. The contradiction above is documented in it, since anyone who goes to check the behaviour against the docs deserves a warning about which page they are going to land on.