Lab 08: The Subnet That Belonged To No Route Table
A new subnet was added alongside an identical working one. Its instances run, pass health checks, answer on their port from inside the VPC, and never register with Systems Manager. Every route table in the account is correct.
- Debugging time
- ~20 min
- Reading time
- 11 min
- Reported by
- Platform Engineering
- Tier
- Associate
Instances in the new subnet never come online in Systems Manager
Reported by Platform Engineering
We added a subnet yesterday for the new worker tier. Same Availability Zone as the existing tier, adjacent CIDR, same security group, same instance profile, same AMI, same launch settings.
Instances come up running and pass both status checks. They never appear in Systems Manager, so
nobody can get a session on them. Their outbound calls time out.
Someone on the team can reach one of them from the existing tier on its application port, so the host itself is clearly fine and the security group is clearly fine.
We have been through every route table in the VPC twice. All three are correct. The internet gateway route is there. We cannot find anything wrong.
What you are working with
One VPC, one Availability Zone, two subnets that were meant to be identical.
Same zone, adjacent CIDRs, both auto-assigning a public IPv4 address, both tagged as public. One of them has a line to the internet gateway and the other does not, and the subnet's own configuration is not what decides that.
- EC2
Worker host, tier B
10.60.12.x — has a public IPv4 address
- SUBNET
Subnet 10.60.12.0/24
no explicit route table association
- RTB
Main route table
local route only — 10.60.0.0/16 → local
Dropped — No route matches a destination outside the VPC, so the packet is discarded at the implicit router. Nothing is denied; there is simply nowhere to send it.
- GW
Internet gateway
attached and healthy — never reached
- DEST
Systems Manager endpoint
ssm.us-east-1.amazonaws.com
The packet dies at the route lookup inside the VPC. The internet gateway is fine and was never involved.
| Resource | Configuration |
|---|---|
| VPC | 10.60.0.0/16, DNS support and hostnames enabled |
| Internet gateway | Attached |
Route table rtb-public | 0.0.0.0/0 → internet gateway, 10.60.0.0/16 → local |
| Main route table | 10.60.0.0/16 → local. Nothing else. |
| Subnet tier A | 10.60.11.0/24, map_public_ip_on_launch |
| Subnet tier B | 10.60.12.0/24, map_public_ip_on_launch |
sg-hosts | Inbound 5000 and ICMP from 10.60.0.0/16, all outbound |
| Instance profile | AmazonSSMManagedInstanceCore on both instances |
Both instances carry the same IAM role, so a permissions explanation can be discarded early.
Scope and constraints
- In scope: why tier B has no path off the VPC.
- Out of scope: the security group, the IAM role, the AMI, and the instance itself. All four are identical to tier A and you should be able to prove it in about two commands.
- The internet gateway is attached and healthy. This is not a detached-gateway bug.
- Tier B instances do receive public IPv4 addresses. That is not the fix and it is part of the trap.
- You will not get an SSM session on tier B. That is the symptom, not an obstacle to work around.
Deploy the broken state
cd lab-08-implicit-route-table-association
terraform init
terraform apply
# Shell on the working host — tier A
aws ssm start-session --target "$(terraform output -raw tier_a_instance_id)"Allow a minute or two after apply. Tier A installs its tooling at launch; tier B cannot, which is itself a clue.
The rest of this lab is part of Labs Pro
The reproduction steps, the deployable Terraform, and the debrief with root cause and remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The incident brief is open, so you can see what you would be solving before you decide.
Already bought it? Sign in and it unlocks.
The rest of this lab is part of Labs Pro
The reproduction steps, the deployable Terraform, and the debrief with root cause and remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The incident brief is open, so you can see what you would be solving before you decide.
Already bought it? Sign in and it unlocks.