Lab 13: The Attachment That Joined The Wrong Route Table

What you are working with#

One Transit Gateway, two VPCs, one Availability Zone.

| Resource | Configuration | | --- | --- | | Transit Gateway | Default association and propagation not specified | | Regulated VPC | 10.110.0.0/16, workload listening on 5000 | | General VPC | 10.111.0.0/16 | | Regulated attachment | Explicitly associated with tgw-rtb-isolation, no propagation | | tgw-rtb-isolation | No routes propagated into it | | General attachment | No explicit association | | General VPC route table | 10.110.0.0/16 → Transit Gateway | | sg-regulated | Port 5000 from 10.110.0.0/16 only; ICMP from 10.0.0.0/8 |

The row that matters is the one that says "no explicit association." Everything else in the table is a deliberate decision that someone made correctly.

Scope and constraints#

• In scope: whether a path exists from the general VPC into the regulated VPC, and which route table decides it.
• Out of scope: the isolation route table's contents. It is correct and it does what it was built to do.
• Both teams' observations are accurate. Reconciling them is the exercise.
• The security group on the regulated host is deliberately narrow. It is not the fix — the question is whether the network boundary holds, not whether a second control catches what gets through.

Deploy the broken state#

cd lab-13-tgw-default-route-table
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw general_instance_id)"

TARGET=$(terraform output -raw regulated_private_ip)

Confirm the path exists#

From the general VPC host:

ping -c 3 "$TARGET"
# 64 bytes from 10.110.1.62: icmp_seq=1 ttl=253 time=1.44 ms
# 3 packets transmitted, 3 received, 0% packet loss

ICMP crosses the Transit Gateway and is answered. Note ttl=253 — two hops consumed, which is what traversing a Transit Gateway looks like.

The security researcher's finding is correct. Now the TCP attempt:

nc -vz -w 5 "$TARGET" 5000
# Ncat: Connection timed out.

That timeout is the security group on the regulated host, not the network. Confirm the difference by watching arrival on the far side.

Watch the traffic arrive inside the regulated VPC#

Open a session on the regulated host and capture:

aws ssm start-session --target "$(terraform output -raw regulated_instance_id)"
sudo tcpdump -ni any "host 10.111.1.0/24 or icmp" -c 6

Then, from the general host, run the ping and the nc again. The regulated host sees:

09:41:12.118 IP 10.111.1.55 > 10.110.1.62: ICMP echo request, id 4210, seq 1
09:41:12.118 IP 10.110.1.62 > 10.111.1.55: ICMP echo reply, id 4210, seq 1
09:41:20.446 IP 10.111.1.55.39820 > 10.110.1.62.5000: Flags [S], seq 118..
09:41:21.449 IP 10.111.1.55.39820 > 10.110.1.62.5000: Flags [S], seq 118..

The SYNs arrive. They are dropped at the interface by the security group, which is why the client sees a timeout, but the packets crossed the Transit Gateway and entered the regulated VPC. A host in the general VPC can address the regulated environment, which means it can port scan it and can deliver UDP payloads to anything with a permissive rule.

Confirm the isolation table really is correct#

The networking team is not wrong either. From the regulated host:

nc -vz -w 5 10.111.1.55 5000
# Ncat: Connection timed out.
 
ping -c 2 10.111.1.55
# 2 packets transmitted, 0 received, 100% packet loss

Nothing. The regulated VPC can initiate nothing across the Transit Gateway, exactly as designed. And the table is genuinely empty:

aws ec2 search-transit-gateway-routes --region us-east-1 \
  --transit-gateway-route-table-id "$(terraform output -raw isolation_route_table_id)" \
  --filters "Name=state,Values=active,blackhole" \
  --query 'Routes[].[DestinationCidrBlock,State]' --output table
# (empty)

Both observations hold. The isolation works in one direction and was only ever capable of working in one direction.

Find which route table the general attachment is in#

aws ec2 get-transit-gateway-route-table-associations --region us-east-1 \
  --transit-gateway-route-table-id "$(terraform output -raw default_route_table_id)" \
  --query 'Associations[].[TransitGatewayAttachmentId,ResourceType,State]' --output table
# --------------------------------------------------------
# |  tgw-attach-0a1b2c3d  |  vpc  |  associated          |
# --------------------------------------------------------

Compare that against the general attachment's ID:

terraform output general_attachment_id
# tgw-attach-0a1b2c3d

It is in the default route table. Now look at what that table can reach:

aws ec2 search-transit-gateway-routes --region us-east-1 \
  --transit-gateway-route-table-id "$(terraform output -raw default_route_table_id)" \
  --filters "Name=state,Values=active" \
  --query 'Routes[].[DestinationCidrBlock,Type,State]' --output table
# ------------------------------------------------------------
# |  10.110.0.0/16  |  propagated  |  active                 |
# |  10.111.0.0/16  |  propagated  |  active                 |
# ------------------------------------------------------------

The regulated prefix is in there, propagated. Nobody added it.

Find why#

aws ec2 describe-transit-gateways --region us-east-1 \
  --transit-gateway-ids "$(terraform output -raw transit_gateway_id)" \
  --query 'TransitGateways[0].Options.{Assoc:DefaultRouteTableAssociation,Prop:DefaultRouteTablePropagation}'
# {
#     "Assoc": "enable",
#     "Prop": "enable"
# }

Both enabled. Neither was ever set in the Terraform.