Subnet Tiering and Route Table Scoping

There is no public = true attribute on a subnet. "Public" and "private" are conventions, not properties. A subnet is public if and only if the route table associated with it has a route to an internet gateway — usually a default route, though a narrower one makes the subnet public for exactly the range it covers. Everything else — naming, tags, the CIDR you chose — is documentation.

Internalizing that one fact resolves most of the confusion around VPC design, because it makes the route table association the real unit of network topology rather than the subnet.

The tier labels are documentation. The line under each subnet is the route table entry that actually decides what the subnet is — and the data tier is isolated only because nothing gave it a default route.

The evaluation order#

When a packet leaves an instance, it is evaluated in a fixed order. Knowing the order tells you where to look when traffic disappears.

1. EC2

   Instance ENI

   Source address must belong to the subnet CIDR
2. FILTER

   Security group — egress rules

   Stateful: the return packet is allowed automatically
3. RTB

   Route table for THIS subnet

   Most specific matching prefix wins, and it decides whether the packet leaves at all
4. FILTER

   Network ACL — outbound rules

   Only on leaving the subnet. Stateless, so the return path needs its own inbound rule
5. GW

   Target of the matched route

   igw, nat, tgw, vpce, eni, or a peering connection

Routing sits between the two filters, not after them. AWS evaluates network ACL rules as traffic enters and leaves a subnet, not as it is routed within one — so whether the ACL is consulted at all is a consequence of the routing decision, and traffic between two instances in the same subnet is never seen by it. Security groups are stateful and NACLs are not, and that asymmetry explains most 'it works one way' bugs.

Two things are worth pausing on.

Security groups are stateful; NACLs are not. If you allow outbound HTTPS in a security group, the response is permitted automatically. A NACL evaluates each direction independently, so blocking inbound ephemeral ports (1024–65535) breaks every outbound connection while looking like a perfectly reasonable inbound restriction.

The tell is a one-way path, not a connection that half works — a TCP connection either completes its handshake or it does not. What you actually see is a timeout: the SYN is permitted, the SYN-ACK matches no rule in the other direction, and the client waits. That precise shape is what Lab 02 is built around.

A security group rule is only as good as your model of the source address. Rules match on what the packet actually carries, and a managed AWS component in the path may or may not replace the source with its own address. An Application Load Balancer terminates the connection and originates a new one, so "allow the load balancer" works. A Network Load Balancer forwards, and on instance target groups it preserves the client address by default — so the same rule permits the health checks and drops every real client, leaving a target group that reports healthy while nothing can connect. That is Lab 06. Before writing a rule that names a source, establish what the target will actually see.

Route selection is longest-prefix-match, and the local route is ordinary in more ways than people expect. A route table holds one route per destination prefix, so you cannot add a competing route for the VPC CIDR alongside the local one — but that is a statement about route tables, not a special protection. Two things follow that are widely believed to be impossible.

You can replace the target of the local route outright. AWS documents replacing or restoring it, and the CLI form is replace-route --destination-cidr-block 10.0.0.0/16 --network-interface-id eni-… — the VPC CIDR pointed at an appliance. A VPC with multiple CIDR blocks has one local route per block, each replaceable on its own.

And you can beat it with a more specific prefix, which is the more common approach and the difference between "impossible" and "supported feature." AWS documents routing intra-VPC traffic through a middlebox appliance: in the source subnet's route table you add a route for the destination subnet's CIDR pointing at the appliance's network interface, alongside the VPC-wide local route. 10.0.16.0/20 → eni-… is longer than 10.0.0.0/16 → local, so longest-prefix-match sends it to the appliance. The local route is untouched; you have simply out-specified it. The console's middlebox routing wizard builds exactly this, and it is worth knowing the shape so you can recognise it — or build it yourself — rather than concluding you need separate VPCs.

There is one real exception, and it is the opposite of what you have just read. A more specific propagated route does not beat the local route: AWS states that where routes propagated from a Site-to-Site VPN or Direct Connect connection overlap the VPC's local route, the local route is preferred even if the propagated routes are more specific. So on-premises advertising a subnet of your VPC's range cannot pull that traffic out of the VPC. The hybrid connectivity guide works through the consequences. Three rules, then, and they are easy to run together: a more specific static route wins, a more specific propagated route loses, and the local target itself can be replaced.

Address planning that survives contact with reality#

Two rules prevent most future pain:

Do not overlap with anything you might one day connect to. Peering refuses outright — AWS will not create a peering connection between VPCs with matching or overlapping CIDRs. A Transit Gateway is more insidious: the attachments succeed, and then its route table holds one route per prefix, so only one of two overlapping VPCs is reachable for that range and nothing reports the other as unreachable. A VPN can be worked around with NAT, at the cost of losing the real source address everywhere. Remediation means renumbering a live VPC, which is a migration, not a change. Assume every VPC will eventually need to reach on-premises and every acquisition will use 10.0.0.0/16.

Leave room in each tier. Subnets cannot be resized after creation, and some consumers set a floor rather than merely filling up: AWS asks that each Availability Zone subnet for an Application Load Balancer have at least a /27 and at least eight free addresses, so that the load balancer can scale out and replace nodes. A /28 public subnet is under that floor from the day it is created, and the symptom when scaling stalls is 5xx errors rather than an error on the subnet.

A layout that holds up for a three-AZ VPC in 10.0.0.0/16:

Every width here is computed from the prefix, so the proportions are real. The public and data tiers are a rounding error against the app tier, and the hatched reserve is what a fourth AZ or a new tier draws from.

At this scale the point the table cannot make becomes obvious: the tiers you think about most are the smallest, the app tier dominates, and well over half the block is still free. That reserve is the difference between adding an AZ and renumbering a live VPC.

Five countable things, so they are drawn as five countable things — a proportional bar cannot say 'eleven, not sixteen'. Under the VPC CNI each pod takes one of these dots, which is why the reserve is a capacity question on EKS rather than a rounding detail: the overhead is a fixed five per subnet, so it hurts most in the small subnets people reach for first.

Both rules are conventions, and neither leaves a trace when it is broken. AWS IPAM is the service that turns them into a registry — with the caveat that its allocation rules report violations rather than prevent them, and that managing private IPv4 space requires a paid tier that bills per address. IPAM pools, compliance, and cost covers what it enforces and what it charges for.

The gap between 10.0.48.0/20 and 10.0.128.0/24 is intentional. It leaves 10.0.64.0/18 — the second quarter of the VPC — entirely unallocated, so a fourth AZ, a new tier, or a secondary CIDR does not force renumbering. Note that the data tier deliberately starts at 10.0.128.0, on the half-way boundary, so the free block is contiguous and easy to reason about rather than scattered between tiers.

One route table per AZ, not one per VPC#

The most common structural mistake in VPC design is a single rtb-private associated with the private subnets in all three AZs.

It looks like sensible deduplication and it forecloses two things you will want:

• Per-AZ egress. A route table holds one route per destination prefix. One shared private route table means one 0.0.0.0/0 target, so all three AZs must egress through the same NAT Gateway — a zonal single point of failure, plus inter-AZ data transfer charges on two thirds of your traffic on top of the NAT Gateway's own data processing charge. Those are two separate line items and they show up under different usage types on the bill.
• Zonal failure isolation. Per-AZ route tables let you steer a single AZ's traffic elsewhere during an impairment. A shared table forces an all-or-nothing change.

Diff of routing.tf: 9 lines added, 3 lines removed.

1		-	# One shared private route table for every AZ
	1	+	# One private route table per AZ, each pointing at that AZ's NAT Gateway.
	2	+	# The route table is the unit of association, so per-AZ egress requires
	3	+	# per-AZ tables — there is no way to express it in a shared one.
2	4		resource "aws_route_table" "private" {
	5	+	for_each = aws_subnet.private
	6	+
3	7		vpc_id = aws_vpc.main.id
4	8
5	9		route {
6	10		cidr_block = "0.0.0.0/0"
7		-	nat_gateway_id = aws_nat_gateway.main.id
	11	+	nat_gateway_id = aws_nat_gateway.per_az[each.key].id
8	12		}
	13	+
	14	+	tags = { Name = "rtb-private-${each.key}" }
9	15		}
10	16
11	17		resource "aws_route_table_association" "private" {
12	18		for_each = aws_subnet.private
13	19
14	20		subnet_id = each.value.id
15		-	route_table_id = aws_route_table.private.id
	21	+	route_table_id = aws_route_table.private[each.key].id
16	22		}

Keying route tables by the same map used for subnets keeps the association one-to-one and makes the per-AZ intent explicit in the plan output.

DNS resolution is scoped to the VPC, not to the network#

Everything above is about moving packets. Name resolution is a separate system with separate scoping rules, and conflating the two produces a failure that survives every network check you can think of.

Each VPC has a Route 53 Resolver reachable at base + 2 — 10.0.0.2 for a 10.0.0.0/16 VPC — and also at the link-local address 169.254.169.253. It is the default nameserver for every instance.

Two VPC attributes gate it, and they do not default the same way, which is the detail that catches people:

Turning the first off is a memorable outage, and rare, because almost nothing works afterwards. The second is the one that bites, because its name undersells it. It reads like a cosmetic setting about public hostnames, and AWS requires both attributes to be true before a private hosted zone or the private DNS name of an interface endpoint will resolve. Since it is false in every VPC you create yourself, a Terraform module that never mentions it produces a VPC where the next two sections of this lesson silently do not work.

The subtler rule is the one worth internalising:

A private hosted zone resolves only in the VPCs it is associated with. Not the VPCs that can reach it. Association is an explicit relationship between a zone and a VPC, and nothing about connectivity creates it.

That means a VPC peering connection, a Transit Gateway attachment, and a VPN all carry packets and carry no DNS. Peer two VPCs and instances on each side can reach the other's addresses immediately, while each side's private zones stay invisible to the other until you associate them. The symptom is NXDOMAIN from a healthy resolver, which is easy to misread as a missing record.

Two consequences that decide how you name zones:

• An associated private zone is authoritative for its whole namespace in that VPC. A name inside the zone's domain with no record returns NXDOMAIN rather than falling through to public DNS. So a private zone for example.com, created to hold three internal records, stops every other example.com name resolving in that VPC — including ones your applications fetch over the internet. Prefer a dedicated subdomain, or a private-only suffix, over a real apex.
• Cross-account association is a two-sided handshake. The zone's account authorises, then the VPC's account associates. Building only the first half looks like cross-account private DNS is unsupported.

Route 53 Resolver endpoints are a different tool and are not the answer to any of the above. They forward queries between a VPC and a network outside AWS — an on-premises resolver, typically — and they cost roughly $0.125 per ENI-hour with a minimum of two. Reaching for them to fix VPC-to-VPC resolution adds about $180 a month and leaves the missing association in place. They belong in the hybrid connectivity picture, not this one.

Lab 05 is built on exactly this: two peered VPCs where the network is provably fine, connecting by IP address works, and the name does not resolve on one side.

The quota that turns route tables into a design constraint#

One route table per AZ reads like a style preference until you know the numbers, at which point it becomes a scaling decision.

Source: Amazon VPC quotas. The static budget is enforced separately for IPv4 and IPv6, and AWS warns that network performance may degrade as you approach the top of it.

The second row is the one to remember, for three reasons. It is an order of magnitude smaller than the first. It cannot be raised at any price. And it is the row you consume without deciding to — propagated routes arrive from a virtual private gateway because someone else's network grew, which is the same organisational failure shape as the advertised-route ceiling in the hybrid connectivity guide. AWS's own guidance when you need more is not a quota request but a design change: advertise a default route instead of every prefix.

Worth being precise about what counts, because it decides whether this row applies to you at all. Propagation into a VPC route table comes from a virtual private gateway — a Site-to-Site VPN or a Direct Connect connection behind one. A route to a Transit Gateway in a VPC route table is a static route and is charged against the 500, not the 100. The propagated budget in a transit gateway's own route tables is a separate thing with separate numbers.

This is also the honest argument for scoping route tables per AZ rather than per VPC. It is not that a shared table runs out of room — 500 routes is a lot. It is that a shared table gives you one 0.0.0.0/0 to spend across every AZ, and no amount of quota fixes that.

One more thing counts against both of these tables and against security groups, and it does not count the way it looks. A route or rule that references a prefix list is charged at the list's maximum size, not the number of entries in it; an AWS-managed list is charged at a published weight, and CloudFront's is 55. So a security group with four visible rules can be two slots from its quota of sixty, and the error that refuses the fifth names neither the rule in the way nor the unit it is counting. Lab 22 is that group.

A checklist that actually finds problems#

When connectivity fails, work the path in order rather than checking your favourite suspect first:

1. Establish the failure mode. Timeout, refusal, TLS error, or DNS failure. This eliminates most of the list immediately.
2. Identify the source subnet, then the route table associated with that specific subnet — not the one you assume applies.
3. Find the matching route by longest prefix, and confirm its target exists and is not blackhole.
4. If the target is a middlebox — NAT Gateway, firewall endpoint, Transit Gateway attachment — repeat steps 2 and 3 for its subnet. This is the step that gets skipped, and it is where Lab 01 lives.
5. Check NACLs in both directions before security groups. Stateless rules break return paths in ways stateful rules cannot, and the deny never appears on either host — only in VPC Flow Logs. Lab 02 is built entirely around this step.

Next#

Client IP preservation takes this per-subnet, per-AZ model into the ingress path, where the question becomes what a target sees in the source field once a load balancer sits in front of it.

After that, NAT Gateway placement and egress cost applies the association model to the egress path, including what the per-gigabyte charges look like when the topology is wrong.