Target Health and Zonal Distribution: Which Zones Receive Traffic

A load balancer's health checks do not only decide which targets receive requests. On a Network Load Balancer they decide which Availability Zones exist as far as DNS is concerned, and that mechanism has a terminal state most people have never looked at: when nothing is healthy anywhere, the load balancer starts sending traffic to everything.

This guide is about where traffic actually lands, which is a different question from whether the load balancer is up.

Click to expand

With cross-zone off, health is reported through DNS: a zone whose target group has no healthy target loses its address. Partial failure shows in the answer; total failure does not, because fail-open publishes every zone again.

Cross-zone is off by default on one and mandatory on the other#

The single most consequential difference between the two load balancer types, and it is a default rather than a feature:

The target group setting wins. If cross-zone is enabled on the load balancer and disabled on the target group, traffic to that target group does not cross zones. One exception: a target group of type alb always inherits from the load balancer and cannot override.

For a Classic Load Balancer the default depends on how you created it — disabled via the API or CLI, enabled via the console — which is worth knowing only because it means you cannot infer the setting from the era of the deployment.

What the distribution actually looks like#

AWS's own worked example is the clearest way to see why this matters. Two Availability Zones, two targets in zone A, eight in zone B. Route 53 hands out load balancer node addresses evenly, so each node receives 50% of client traffic.

With cross-zone off, each node can only reach targets in its own zone, so zone A's two targets split the same 50% that zone B's eight targets split. A four-to-one imbalance in instance count becomes a four-to-one imbalance in load per instance.

Every target is drawn, so the four-to-one difference in count is the picture rather than a figure in a table. Route 53 hands out the two node addresses evenly, so each column receives half the clients regardless of what is behind it — and with cross-zone off a node can only reach its own column. Two targets splitting half the traffic each get four times the load of eight targets splitting the other half.

This is the practical argument for symmetric capacity per zone rather than symmetric capacity in total. An autoscaling group that is "big enough" across the Region can still be badly overloaded in one zone.

Zonal DNS is the mechanism#

A Network Load Balancer has a DNS record and an IP address per enabled Availability Zone. Health does not just gate targets, it gates those records:

• Every enabled zone with at least one healthy target publishes its address.
• A zone whose target group has no healthy target has its address removed from DNS, so new connections stop being directed there at all. AWS's zonal health and health check pages both state this without qualification; the troubleshooting page adds "and cross-zone load balancing is disabled", which is the reading that makes mechanical sense — a node with cross-zone on can still reach healthy targets elsewhere, so there is nothing to withdraw for. Do not count on zone withdrawal as a failure signal while cross-zone is on.
• Zonal health is visible as the ZonalHealthStatus CloudWatch metric, which is the signal you want to alarm on rather than inferring zone withdrawal from a drop in traffic.

A zone can fail its zonal health check for more reasons than "no healthy targets" — the healthy count falling below a configured minimum, a zonal shift or auto-shift in progress, or AWS shifting traffic away from a zone it has detected problems in. Those last ones mean a zone can disappear from DNS without anything being wrong with your targets.

Fail-open makes total failure look like total health#

Here is the part worth internalising. If every target fails health checks in every enabled zone, the load balancer does not stop serving. It fails open: DNS publishes all zonal addresses again, and traffic is sent to all targets regardless of their health status.

A Network Load Balancer also fails open when the target group is empty — and the two facts are one mechanism, not two. An NLB reports an empty target group as unhealthy; with every target unhealthy, the all-unhealthy fail-open rule applies, and every zonal address is published. An ALB does not take that path, because it does not report an empty target group as unhealthy in the first place. The next two sections return to that difference.

So consider what an outside observer sees. A fully healthy load balancer publishes every zonal address. A completely broken one publishes every zonal address. Partial failure is the only state where DNS tells you something is wrong, because that is the only state where an address is missing.

While partial failure is in effect, a target that has gone unhealthy gets a TCP RST for packets on client connections associated with it, rather than a silent drop. That is a useful distinction when reading a client-side trace: a reset points at a target the load balancer has already given up on, a timeout points somewhere else.

Health check details that cause false failures#

Five behaviours that produce unhealthy targets for reasons unrelated to the application:

• A redirect is a failed health check. The check compares the response code to the Matcher, whose default is 200, and it does not follow a Location header. An application hardened to redirect HTTP to HTTPS answers the health check with 301, is marked Target.ResponseCodeMismatch, and goes out of service for doing what the security review asked. AWS's own worked example of that reason code in the health check logs is a 301. Lab 21 is this failure, and the design decision — exempt the health path, widen the matcher, or check HTTPS — is what it is about.

• TLS 1.3-only targets fail HTTPS health checks. A target group configured with HTTPS health checks requires the target to support an earlier version such as TLS 1.2. Hardening a target to TLS 1.3 only will take it out of service.

• The health check Host header is not the target. For HTTP and HTTPS health checks it contains the IP address of the load balancer node and the listener port, not the target's address and health check port. Virtual-host routing or a strict Host check on the target will reject it.

• A TLS listener generates extra connections. Adding one triggers a listener connectivity test, so you will see TCP connections from the load balancer to targets that carry no data packets. They are not an attack and not a misconfiguration.

• An empty target group is read differently by the two load balancers. AWS's DNS troubleshooting guidance states the asymmetry directly: "With Application Load Balancers, an empty target group might not report an error and therefore can appear healthy. […] With Network Load Balancers, an empty target group is reported as unhealthy." So on an NLB an empty zone is withdrawn from DNS like any unhealthy one; on an ALB it stays in DNS looking fine and serves 503s, which is the case the next section is about.

Turning cross-zone off has prerequisites#

If you disable it for zonal isolation or latency, the constraints are real:

• Provision enough target capacity in every zone the load balancer is enabled in.
• Ensure all target groups participate in the same zones. AWS's target group attributes page gives the reason: a zone that is empty for one target group while cross-zone is off "triggers a 503 error for all HTTP requests that enter the empty Availability Zone."
• On an Application Load Balancer, target stickiness is not supported with cross-zone off, and neither are Lambda targets. Registering targets also requires the AvailabilityZone parameter, and specific zone values are only accepted when cross-zone is off.
• On an Application Load Balancer, avoid empty subnets entirely. From the same page: "Application Load Balancers expose zonal IP addresses through DNS for the empty subnets, which triggers 503 errors for HTTP requests."

That last one is the trap in an otherwise reasonable change: adding a subnet in a new zone ahead of the capacity that will live there creates a zone that answers DNS and serves errors. It is also the ALB half of the asymmetry above — an NLB would withdraw the empty zone, and an ALB keeps answering for it.

What to instrument#

• ZonalHealthStatus, per zone. This is the direct signal that a zone has left DNS, and it is more precise than watching request counts fall.
• Healthy target count per zone, with a threshold above one. The interesting alarm is not zero — zero in every zone triggers fail-open and the symptom changes shape. It is one, in a zone that should have several.
• Whether fail-open is currently active. All-zones-unhealthy is a state you want paged on, precisely because DNS looks normal during it.
• Target group emptiness. An empty target group fails open, which means a failed deployment that deregistered everything presents as a load balancer distributing traffic to nothing in particular.

Next#

The egress track picks up the other direction, including the per-AZ data processing charges that cross-zone load balancing contributes to.