NAT Gateway Placement, Failure Modes, and What Egress Actually Costs
Where a NAT Gateway lives determines whether it works at all, how it fails when an AZ degrades, and how much of your data transfer bill is avoidable. All three answers come from the same design decision.
Published Feb 6, 2026
A NAT Gateway is one of the few AWS networking primitives where a single attribute — which subnet you put it in — simultaneously decides whether traffic flows, what happens during an AZ impairment, and how much you pay per gigabyte. Most teams get the first part right by copying a reference architecture and never revisit the other two.
Follow the arrows upward. The private tier's default route reaches the NAT Gateway, and the NAT Gateway then forwards using the route table of the subnet it sits in. Nothing reaches the internet gateway unless that second table has a route to it, which is the entire reason placement decides whether egress works. The gateway is marked ↗ because it opens the lab that puts it in the wrong subnet.
What a NAT Gateway actually is
It helps to be precise, because the mental model most people carry is wrong in a way that produces real outages.
A NAT Gateway is not a service endpoint that you point traffic at and it reaches the internet. It is a managed appliance that occupies an IP address in a specific subnet. When a packet arrives, it performs source network address translation and then forwards the result according to the route table associated with its own subnet.
That last clause is the whole thing. Two consequences follow:
- A NAT Gateway must sit in a subnet whose route table has a default route to an internet gateway. In practice that means a public subnet.
- A NAT Gateway placed in the private subnet it serves creates a routing loop. Traffic arrives, gets translated, and is handed back to the same route table that just sent it there.
The failure signature is a timeout, never a refusal. VPC routing drops packets without generating ICMP unreachable messages, so the client retransmits SYNs until it gives up. If you remember one diagnostic heuristic from this guide, make it that one:
| Symptom | What it eliminates |
|---|---|
| Connection refused | Routing is fine — you reached a host and nothing was listening on that port |
| Connection timed out | The packet died in transit — routing, a NACL, or a security group |
| TLS or certificate error | Routing and TCP are both fine — the problem is above layer 4 |
| Name resolution failure | Never reached the routing layer at all — no packet was ever addressed |
A timeout on outbound traffic from a private subnet means you are looking for something that forwards. Start with the middlebox, not the client.
Reading the two route tables together
Diagnosing this class of problem requires looking at two route tables as a pair. Neither is wrong on its own.
| Destination | Target | State | Origin |
|---|---|---|---|
| 10.0.0.0/16 | local | active | CreateRouteTable |
| 0.0.0.0/0 | nat-0f8e… | activeCorrect — the private tier should egress via NAT. Nothing to fix here. | CreateRoute |
The private route table is exactly right, which is why teams stop looking after checking it.
| Destination | Target | State | Origin |
|---|---|---|---|
| 10.0.0.0/16 | local | active | CreateRouteTable |
| 0.0.0.0/0 | igw-04c… | activeThis is the route the NAT Gateway needs to consult. It only does so if it is in THIS subnet. | CreateRoute |
Both tables are individually valid. The defect is the relationship between them and the NAT Gateway's placement.
This is the most common shape of a cloud networking bug: every object is healthy, and the fault is
in a relationship no single describe call reveals. When a checklist of individual resources all
comes back green and traffic still fails, widen the aperture instead of re-checking the same
resources harder.
Availability: one NAT Gateway per AZ
A zonal NAT Gateway is exactly that: it lives in one Availability Zone and it does not fail over.
A single shared NAT Gateway serving private subnets across three AZs produces two problems:
- Blast radius. If that AZ is impaired, egress breaks for all three AZs, not one. You have taken a zonal dependency for your entire outbound path.
- Cross-AZ data processing charges. Traffic from
us-east-1bto a NAT Gateway inus-east-1acrosses an AZ boundary and is billed for it, on top of the NAT processing charge.
The hole in the public row is the whole problem: AZ b and AZ c have a public subnet but no gateway of their own, so two thirds of egress crosses an AZ boundary to reach AZ a and the entire VPC's outbound path depends on one zone.
With zonal gateways the correct pattern is one NAT Gateway per AZ, each in that AZ's public subnet, with each private subnet's route table pointing at the gateway in its own AZ. This is the work a regional NAT Gateway does for you: one gateway ID that every private subnet can route to, expanding into a zone when it sees a network interface there and contracting when the zone empties.
What it costs
NAT Gateway pricing has two components, and the second is the one that surprises people:
- An hourly charge per gateway, roughly $0.045/hour in
us-east-1. About $32/month, billed whether or not a single packet flows. A misconfigured, non-functional NAT Gateway bills exactly the same as a working one. - A data processing charge per gigabyte, roughly $0.045/GB, applied to all traffic through the gateway in either direction.
That per-GB charge is where budgets quietly break. It is levied in addition to any internet data transfer cost, and it applies to traffic that never leaves AWS.
The S3 case
Consider an application in a private subnet pulling 10 TB/month from S3 in the same region.
Routed through a NAT Gateway, that 10 TB incurs the data processing charge — roughly $450/month — even though the traffic never touches the internet. A gateway VPC endpoint for S3 costs nothing per hour and nothing per gigabyte, and removes the NAT Gateway from that path entirely.
The per-gigabyte charge is fourteen times the hourly one at this volume, and it is the line people never model. Drawn against each other the priority is obvious: the gateway is not the expensive part, the traffic you sent through it is. The third bar is a hairline rather than nothing, because free and absent are different claims.
The fix is a route, not an application change:
| 1 | - | # S3 traffic from private subnets egresses via NAT | |
| 2 | 1 | resource "aws_route_table" "private" { | |
| 3 | 2 | vpc_id = aws_vpc.main.id | |
| 4 | 3 | ||
| ⋯ 2 unchanged lines | |||
| 7 | 6 | nat_gateway_id = aws_nat_gateway.per_az["a"].id | |
| 8 | 7 | } | |
| 9 | 8 | } | |
| 9 | + | ||
| 10 | + | # Gateway endpoints are free: no hourly charge, no per-GB charge. Associating | |
| 11 | + | # one injects a prefix-list route that bypasses the NAT Gateway for S3. | |
| 12 | + | resource "aws_vpc_endpoint" "s3" { | |
| 13 | + | vpc_id = aws_vpc.main.id | |
| 14 | + | service_name = "com.amazonaws.${var.region}.s3" | |
| 15 | + | vpc_endpoint_type = "Gateway" | |
| 16 | + | route_table_ids = [aws_route_table.private.id] | |
| 17 | + | } | |
Gateway endpoints exist for S3 and DynamoDB only. Everything else is an interface endpoint, which does carry an hourly charge.
Gateway versus interface endpoints
The distinction matters for cost modelling:
- Gateway endpoints (S3, DynamoDB) are free. They work by injecting a prefix-list route into the route tables you associate. Take them for in-VPC traffic — but they are not a universal answer: a gateway endpoint is reachable only from inside the VPC it is associated with, so it does not serve on-premises networks or other Regions. If Direct Connect or a peered Region needs the same path, you want an interface endpoint. Two further caveats bite in practice: traffic still addresses the service's public endpoint, so security groups and NACLs must permit it — reference the service's prefix list rather than hardcoding ranges — and an endpoint policy that is more restrictive than you expect is the usual reason a rollout stalls.
- Interface endpoints (everything else — SSM, ECR, Secrets Manager, and so on) are ENIs in your subnets, billed hourly per endpoint per AZ plus a per-GB charge. Cheaper than NAT for high-volume paths, more expensive for trivial ones. Model it before deploying dozens.
The two ways a healthy NAT Gateway starts failing
Everything above is about placement and cost. Both of those are design-time problems. The two failures that show up later, on a NAT Gateway nobody has touched, are worth knowing before they happen — and neither produces a timeout, which is what makes them confusing after Lab 01 trains you to expect one.
Port exhaustion
Each IPv4 address on a NAT Gateway supports 55,000 simultaneous connections to each unique destination, where a unique destination is the combination of destination IP address, destination port, and protocol. Note that the budget belongs to the address, not the gateway — which is what makes the remedy below work.
The per-destination part is the whole point. 55,000 connections spread across a thousand different API endpoints is nothing. 55,000 connections to one address and port is a ceiling you can hit with a single busy service — a shared database behind a private endpoint, a payment provider, an internal API that every pod calls. Fan-out saves you; concentration kills you. Which means the metric to watch is not total connection count.
That metric is ErrorPortAllocation — the number of times the NAT Gateway could not allocate a
source port. It should be flat zero, so any non-zero value is actionable, and it is a far better
signal than the cost alarm below because it fires while the failure is still partial.
The remedy is more source addresses. A zonal NAT Gateway can carry up to 8 IPv4 addresses (one primary plus seven secondary), and each one gets its own 55,000-per-destination budget. Note the wrinkle: a public NAT Gateway is limited to 2 Elastic IPs by default, so going beyond that is a quota request rather than a configuration change — which is a bad thing to discover during the incident.
A regional NAT Gateway raises the ceiling substantially: 32 IP addresses per Availability Zone rather than 8 for the whole gateway. If you are hitting port allocation errors against a concentrated destination, that headroom is a better argument for regional mode than the availability story is.
IdleTimeoutCount is the leading indicator here. AWS recommends watching it for rising idle connections
and closing them to release capacity, which is the cheapest way to buy headroom before you start
requesting address quota.
The 350-second idle timeout
A NAT Gateway drops a connection after 350 seconds of idleness, and then returns an RST to anything behind it that tries to continue that connection. AWS is specific that it does not send a FIN.
The timing matters: nothing arrives at second 350. The connection is simply gone, and your client learns about it the next time it writes — which can be minutes later, and looks like the connection died mid-use rather than while idle.
Read that symptom carefully, because it inverts the diagnostic table at the top of this guide. A connection reset normally tells you that you reached a host and it refused you. Here it tells you the opposite: you reached the host fine, held an idle connection too long, and the middlebox in the path had already discarded it. Long-lived database pools and idle worker connections are the usual victims, and the fix is on the client — TCP keepalives below 350 seconds — not on the gateway, which offers no knob for this.
When one NAT Gateway per AZ stops being the answer
The advice above terminates at one NAT Gateway per Availability Zone. That is correct for a VPC. It is a poor answer for thirty, because you are now paying an hourly charge per AZ per VPC for capacity that is mostly idle — ninety gateways to give thirty VPCs three-AZ egress.
There are two steps, and most teams skip straight to the second. The first is regional mode, which collapses three gateways per VPC into one resource and removes the per-AZ route tables — the hourly rate is still charged per AZ, so it saves operational surface rather than money. The second is centralized egress: one egress VPC holding the NAT Gateways, every other VPC attached to a Transit Gateway with a default route pointing at it. That is the one that changes the bill, because you trade the per-VPC hourly cost for TGW attachment hours and per-GB processing — arithmetic rather than doctrine, worth modelling once you are past roughly a dozen VPCs and worth ignoring below that. Regional gateways accept a Transit Gateway as a route target in their own route table, so the two are compatible.
Two constraints decide whether it is even available to you:
- You cannot route to a NAT Gateway through a VPC peering connection. Peering does not carry transitive traffic to a middlebox, so a peered hub-and-spoke cannot do this. It has to be a Transit Gateway.
- You cannot route to a NAT Gateway from a Site-to-Site VPN or Direct Connect via a virtual private gateway — but you can via a Transit Gateway. So if on-premises traffic needs to egress through your AWS NAT, the choice of hybrid endpoint made back in the hybrid connectivity guide has already decided this for you.
The egress VPC itself has two route tables whose default routes must differ: the attachment subnet
sends 0.0.0.0/0 to the NAT Gateway, and the NAT Gateway's subnet sends it to the internet gateway.
Copy one onto the other and spoke traffic reaches the internet gateway untranslated, which discards it
without a log line anywhere. Lab 20 is that failure, built to the
reference architecture with one target wrong.
If you build it, appliance mode on the egress VPC attachment is what keeps a flow pinned to one AZ. That is covered in the Transit Gateway routing guide, including the propagation prerequisite that quietly disables it.
Guardrails
Placement is not something to leave to review. Four checks worth automating:
- A policy assertion that
aws_nat_gateway.subnet_idresolves to a subnet taggedTier = "public". Cheap intflint, Sentinel, or Conftest, and it catches the outage class described above before apply. - A synthetic egress check from the private tier in CI. This failure mode is invisible until real traffic needs it — which, if you are unlucky, is during a batch run at month end.
- A cost alarm on
NatGateway-Bytes. A sudden step change usually means a new workload started pulling from S3 or ECR through NAT instead of through an endpoint. - An alarm on
ErrorPortAllocation > 0. The only one of these four that catches a live degradation rather than a design mistake, and the threshold is genuinely zero.
Where to practice
The failure mode at the top of this guide is Lab 01. It deploys a VPC where the NAT Gateway reports healthy and every outbound connection times out. Work the ticket before reading the debrief.