AssociateProAWS VPC

Lab 08: The Subnet That Belonged To No Route Table

A new subnet was added alongside an identical working one. Its instances run, pass health checks, answer on their port from inside the VPC, and never register with Systems Manager. Every route table in the account is correct.

Debugging time: ~20 min
Reading time: 11 min
Reported by: Platform Engineering
Tier: Associate

INC-1455SEV-3InvestigatingOpened 2026-09-17 08:40 UTC

Instances in the new subnet never come online in Systems Manager

Reported by Platform Engineering

We added a subnet yesterday for the new worker tier. Same Availability Zone as the existing tier, adjacent CIDR, same security group, same instance profile, same AMI, same launch settings.

Instances come up running and pass both status checks. They never appear in Systems Manager, so nobody can get a session on them. Their outbound calls time out.

Someone on the team can reach one of them from the existing tier on its application port, so the host itself is clearly fine and the security group is clearly fine.

We have been through every route table in the VPC twice. All three are correct. The internet gateway route is there. We cannot find anything wrong.

What you are working with

One VPC, one Availability Zone, two subnets that were meant to be identical.

EC2
Worker host, tier B
10.60.12.x — has a public IPv4 address
SUBNET
Subnet 10.60.12.0/24
no explicit route table association
RTB
Main route table
local route only — 10.60.0.0/16 → local
Dropped — No route matches a destination outside the VPC, so the packet is discarded at the implicit router. Nothing is denied; there is simply nowhere to send it.
GW
Internet gateway
attached and healthy — never reached
DEST
Systems Manager endpoint
ssm.us-east-1.amazonaws.com

The packet dies at the route lookup inside the VPC. The internet gateway is fine and was never involved.

| Resource | Configuration | | --- | --- | | VPC | 10.60.0.0/16, DNS support and hostnames enabled | | Internet gateway | Attached | | Route table rtb-public | 0.0.0.0/0 → internet gateway, 10.60.0.0/16 → local | | Main route table | 10.60.0.0/16 → local. Nothing else. | | Subnet tier A | 10.60.11.0/24, map_public_ip_on_launch | | Subnet tier B | 10.60.12.0/24, map_public_ip_on_launch | | sg-hosts | Inbound 5000 and ICMP from 10.60.0.0/16, all outbound | | Instance profile | AmazonSSMManagedInstanceCore on both instances |

Both instances carry the same IAM role, so a permissions explanation can be discarded early.

Scope and constraints

In scope: why tier B has no path off the VPC.
Out of scope: the security group, the IAM role, the AMI, and the instance itself. All four are identical to tier A and you should be able to prove it in about two commands.
The internet gateway is attached and healthy. This is not a detached-gateway bug.
Tier B instances do receive public IPv4 addresses. That is not the fix and it is part of the trap.
You will not get an SSM session on tier B. That is the symptom, not an obstacle to work around.

Deploy the broken state

cd lab-08-implicit-route-table-association
terraform init
terraform apply
 
# Shell on the working host — tier A
aws ssm start-session --target "$(terraform output -raw tier_a_instance_id)"

Allow a minute or two after apply. Tier A installs its tooling at launch; tier B cannot, which is itself a clue.

Collect the identifiers first. Use the outputs rather than the sample values below, since addresses and IDs differ on every apply.

terraform output tier_b_instance_id
terraform output tier_b_private_ip
terraform output tier_a_subnet_id
terraform output tier_b_subnet_id

Confirm the symptom from the API

Before touching a host, establish that tier B never registered.

aws ssm describe-instance-information \
  --query 'InstanceInformationList[].InstanceId' --output json
# [
#     "i-0aa11bb22cc33dd44"
# ]

One instance, not two. And the instance itself is unambiguously healthy:

aws ec2 describe-instance-status \
  --instance-ids "$(terraform output -raw tier_b_instance_id)" \
  --query 'InstanceStatuses[0].[InstanceState.Name,InstanceStatus.Status,SystemStatus.Status]' \
  --output text
# running	ok	ok

Running, both checks ok, and absent from Systems Manager. A host that boots correctly but cannot reach an AWS service endpoint is a networking problem, not a host problem.

Prove the host is alive from inside the VPC

From the tier A session. This is the step that eliminates most of the checklist.

B=10.60.12.87   # terraform output -raw tier_b_private_ip
 
ping -c 3 "$B"
# 3 packets transmitted, 3 received, 0% packet loss
 
nc -w 3 "$B" 5000
# lab-08-tier-b-ok

The host is up, the listener is running, the security group permits the traffic, and in-VPC routing works in both directions. Whatever is broken applies only to destinations outside the VPC.

Confirm the direction of the failure

Still from tier A, which does have egress:

curl -s -m 5 -o /dev/null -w '%{http_code}\n' https://ssm.us-east-1.amazonaws.com/
# 404

A 404 here is success — the endpoint answered. Tier A can reach the Systems Manager endpoint, from the same VPC, in the same Availability Zone, through the same security group.

So the difference between the two hosts is not the host, not the SG, and not the gateway. It is the subnet.

Read the route tables the way the ticket did

VPC=$(aws ec2 describe-subnets --subnet-ids "$(terraform output -raw tier_a_subnet_id)" \
        --query 'Subnets[0].VpcId' --output text)
 
aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC" \
  --query 'RouteTables[].[RouteTableId,Associations[0].Main,join(`,`,Associations[?SubnetId].SubnetId || [`none`])]' \
  --output table
# ------------------------------------------------------------
# |  rtb-0f1e2d3c   |  True   |  none                        |
# |  rtb-0a9b8c7d   |  None   |  subnet-0aaa111  (tier A)    |
# ------------------------------------------------------------

Read that carefully. The main route table has no subnet associations. The public route table has exactly one: tier A.

Tier B's subnet does not appear in either list.

Ask the question directly

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=$(terraform output -raw tier_a_subnet_id)" \
  --query 'RouteTables[].RouteTableId' --output text
# rtb-0a9b8c7d
 
aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=$(terraform output -raw tier_b_subnet_id)" \
  --query 'RouteTables[].RouteTableId' --output text
# (empty)

Tier A resolves to a route table. Tier B resolves to nothing at all.

Instances in the new subnet never come online in Systems Manager

This debrief is part of Labs Pro