ProfessionalFree accountTransit GatewayAWS VPC

Lab 03: The Spoke That Could Not Be Answered

Two spoke VPCs built from the same Terraform module. One reaches shared services, the other times out. Every attachment is available, both VPC route tables are identical, and the Transit Gateway shows both spokes correctly associated.

Debugging time: ~30 min
Reading time: 13 min
Reported by: Platform Networking
Tier: Professional

INC-1142SEV-2InvestigatingOpened Today 10:47 UTC

Newly onboarded spoke VPC cannot reach shared services; the existing spoke works

Reported by Platform Networking

Environment: staging
Region: us-east-1
Working spoke: spoke-a — 10.1.0.0/16
Broken spoke: spoke-b — 10.2.0.0/16
Target: shared services — 10.0.0.0/16

We onboarded spoke-b yesterday using the same Terraform module that built spoke-a. The internal config API in shared services is reachable from spoke-a and times out from spoke-b.

The two VPCs are as close to identical as two VPCs get:

Same module, same version, only the CIDR inputs differ.
Both VPC route tables send 10.0.0.0/8 at the Transit Gateway. We diffed them.
Both Transit Gateway attachments report available.
Both attachments are associated with the tgw-rt-spokes route table. We checked this specifically because it is the usual culprit.
Security groups on the shared services host allow the service port from 10.0.0.0/8, which covers both spokes.

The onboarding runbook has three steps and we completed all three. Networking has looked at it for two hours. We are not sure what is left to check.

What you are working with

A standard hub-and-spoke Transit Gateway topology. Two Transit Gateway route tables implement the policy: spokes may reach shared services, spokes may not reach each other.

Starting state — the VPC layer is identical in all three networks

There is one transit gateway; it is drawn on each path it carries. Every VPC is built from the same definition — same summary route at 10.0.0.0/8, same attachment, same association — and each also holds a 0.0.0.0/0 route to its own internet gateway so the hosts can install tooling. Nothing at this layer distinguishes the spoke that works from the one that does not, which is why the answer is not in this picture.

10.2.1.20:52104 → 10.0.1.20:8080 (TCP SYN, spoke-b to shared)

EC2
spoke-b host
10.2.1.20 in vpc spoke-b (10.2.0.0/16)
RTB
spoke-b VPC route table
10.0.0.0/8 → tgw-…
RTB
TGW route table: tgw-rt-spokes
consulted because the spoke-b attachment is associated here
GW
shared services attachment
10.0.0.0/16 is propagated into tgw-rt-spokes
DEST
shared services host
10.0.1.20:8080 — listening

The request path is complete and correct. As in Lab 02, the interesting direction is the one nobody drew.

Scope and constraints

In scope: Transit Gateway route tables, associations, and propagations.
Out of scope: security groups, NACLs, VPC route tables, DNS, and the service itself. All are correct and identical across both spokes.
spoke-a cannot reach spoke-b, and that is intentional. Spoke-to-spoke isolation is the point of this topology. Do not chase it as a second bug — but do be able to explain why it fails, because it is the same mechanism as the real defect.
You get a shell on all three hosts. Every host runs the same listener on port 8080, so you can probe in any direction.

Deploy the broken state

main.tf8 lines

Download

Download the file below into an empty directory and apply it. Expect four to six minutes — Transit Gateway attachments are the slow part.

terraform init
terraform apply

# Shells for all three hosts
terraform output start_session_commands

# Private IPs to probe
terraform output private_ips

Full source: main.tf. Three VPCs generated from one for_each, so the VPC layer is provably symmetric, plus the Transit Gateway, two route tables, and the association and propagation resources.

Grab the addresses first:

terraform output private_ips
# {
#   "shared"  = "10.0.1.20"
#   "spoke-a" = "10.1.1.20"
#   "spoke-b" = "10.2.1.20"
# }

Establish the asymmetry

From spoke-a:

aws ssm start-session --region us-east-1 --target <spoke-a instance id>
 
curl -sS -m 8 http://10.0.1.20:8080/
# vn-lab-03 shared ok

From spoke-b:

aws ssm start-session --region us-east-1 --target <spoke-b instance id>
 
curl -sS -m 8 http://10.0.1.20:8080/
# curl: (28) Connection timed out after 8001 milliseconds

Same command, same target, same module. One works.

Confirm the request actually arrives

This is the step that reframes the problem. Open a shell on shared services and capture while you retry from spoke-b:

# On the shared services host
sudo tcpdump -ni any 'tcp port 8080' &
 
# From spoke-b, in another session
curl -sS -m 8 http://10.0.1.20:8080/

On the shared host you will see something like:

10:58:11.204 IP 10.2.1.20.52104 > 10.0.1.20.8080: Flags [S], seq 1104...
10:58:11.204 IP 10.0.1.20.8080 > 10.2.1.20.52104: Flags [S.], seq 3392...
10:58:12.210 IP 10.2.1.20.52104 > 10.0.1.20.8080: Flags [S], seq 1104...
10:58:12.210 IP 10.0.1.20.8080 > 10.2.1.20.52104: Flags [S.], seq 3392...

The SYN arrives and the SYN-ACK goes out. The request is fine. The reply is being lost on the way back, somewhere between the shared services VPC and spoke-b.

Rule out what the ticket already checked

Attachment states:

aws ec2 describe-transit-gateway-attachments \
  --query 'TransitGatewayAttachments[].[TransitGatewayAttachmentId,ResourceId,State]' \
  --output table

All three available. Now the associations — the thing the runbook checks:

SPOKES=$(terraform output -json tgw_route_table_ids | jq -r .spokes)
SHARED=$(terraform output -json tgw_route_table_ids | jq -r .shared)
 
aws ec2 get-transit-gateway-route-table-associations \
  --transit-gateway-route-table-id "$SPOKES" \
  --query 'Associations[].[TransitGatewayAttachmentId,State]' --output table

Both spoke attachments are associated with tgw-rt-spokes, exactly as the ticket says. The runbook step was completed correctly.

Look at the route tables themselves

An association tells you which route table an attachment consults. It says nothing about what is in that route table. Look at the contents:

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id "$SHARED" \
  --filters "Name=state,Values=active,blackhole" \
  --query 'Routes[].[DestinationCidrBlock,Type,State]' --output table

Then ask the targeted question — can this route table reach the spoke-b host at all?

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id "$SHARED" \
  --filters "Name=route-search.longest-prefix-match,Values=10.2.1.20" \
  --query 'Routes' --output json
# []

An empty result is the answer. Now compare what each route table has learned:

for RT in "$SPOKES" "$SHARED"; do
  echo "--- $RT"
  aws ec2 get-transit-gateway-route-table-propagations \
    --transit-gateway-route-table-id "$RT" \
    --query 'TransitGatewayRouteTablePropagations[].[TransitGatewayAttachmentId,ResourceId,State]' \
    --output table
done

Newly onboarded spoke VPC cannot reach shared services; the existing spoke works

What you are working with#

Scope and constraints#

Deploy the broken state#

Establish the asymmetry#

Confirm the request actually arrives#

Rule out what the ticket already checked#

Look at the route tables themselves#

Sign in to read the debrief