Lab 03: The Spoke That Could Not Be Answered

What you are working with#

A standard hub-and-spoke Transit Gateway topology. Two Transit Gateway route tables implement the policy: spokes may reach shared services, spokes may not reach each other.

Scope and constraints#

• In scope: Transit Gateway route tables, associations, and propagations.
• Out of scope: security groups, NACLs, VPC route tables, DNS, and the service itself. All are correct and identical across both spokes.
• spoke-a cannot reach spoke-b, and that is intentional. Spoke-to-spoke isolation is the point of this topology. Do not chase it as a second bug — but do be able to explain why it fails, because it is the same mechanism as the real defect.
• You get a shell on all three hosts. Every host runs the same listener on port 8080, so you can probe in any direction.

Deploy the broken state#

terraform init
terraform apply

# Shells for all three hosts
terraform output start_session_commands

# Private IPs to probe
terraform output private_ips

Full source: main.tf. Three VPCs generated from one for_each, so the VPC layer is provably symmetric, plus the Transit Gateway, two route tables, and the association and propagation resources.

Grab the addresses first:

terraform output private_ips
# {
#   "shared"  = "10.0.1.20"
#   "spoke-a" = "10.1.1.20"
#   "spoke-b" = "10.2.1.20"
# }

Establish the asymmetry#

From spoke-a:

aws ssm start-session --region us-east-1 --target <spoke-a instance id>
 
curl -sS -m 8 http://10.0.1.20:8080/
# vn-lab-03 shared ok

From spoke-b:

aws ssm start-session --region us-east-1 --target <spoke-b instance id>
 
curl -sS -m 8 http://10.0.1.20:8080/
# curl: (28) Connection timed out after 8001 milliseconds

Same command, same target, same module. One works.

Confirm the request actually arrives#

This is the step that reframes the problem. Open a shell on shared services and capture while you retry from spoke-b:

# On the shared services host
sudo tcpdump -ni any 'tcp port 8080' &
 
# From spoke-b, in another session
curl -sS -m 8 http://10.0.1.20:8080/

On the shared host you will see something like:

10:58:11.204 IP 10.2.1.20.52104 > 10.0.1.20.8080: Flags [S], seq 1104...
10:58:11.204 IP 10.0.1.20.8080 > 10.2.1.20.52104: Flags [S.], seq 3392...
10:58:12.210 IP 10.2.1.20.52104 > 10.0.1.20.8080: Flags [S], seq 1104...
10:58:12.210 IP 10.0.1.20.8080 > 10.2.1.20.52104: Flags [S.], seq 3392...

The SYN arrives and the SYN-ACK goes out. The request is fine. The reply is being lost on the way back, somewhere between the shared services VPC and spoke-b.

Rule out what the ticket already checked#

Attachment states:

aws ec2 describe-transit-gateway-attachments \
  --query 'TransitGatewayAttachments[].[TransitGatewayAttachmentId,ResourceId,State]' \
  --output table

All three available. Now the associations — the thing the runbook checks:

SPOKES=$(terraform output -json tgw_route_table_ids | jq -r .spokes)
SHARED=$(terraform output -json tgw_route_table_ids | jq -r .shared)
 
aws ec2 get-transit-gateway-route-table-associations \
  --transit-gateway-route-table-id "$SPOKES" \
  --query 'Associations[].[TransitGatewayAttachmentId,State]' --output table

Both spoke attachments are associated with tgw-rt-spokes, exactly as the ticket says. The runbook step was completed correctly.

Look at the route tables themselves#

An association tells you which route table an attachment consults. It says nothing about what is in that route table. Look at the contents:

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id "$SHARED" \
  --filters "Name=state,Values=active,blackhole" \
  --query 'Routes[].[DestinationCidrBlock,Type,State]' --output table

Then ask the targeted question — can this route table reach the spoke-b host at all?

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id "$SHARED" \
  --filters "Name=route-search.longest-prefix-match,Values=10.2.1.20" \
  --query 'Routes' --output json
# []

An empty result is the answer. Now compare what each route table has learned:

for RT in "$SPOKES" "$SHARED"; do
  echo "--- $RT"
  aws ec2 get-transit-gateway-route-table-propagations \
    --transit-gateway-route-table-id "$RT" \
    --query 'TransitGatewayRouteTablePropagations[].[TransitGatewayAttachmentId,ResourceId,State]' \
    --output table
done

Root cause#

The spoke-b attachment was never propagated into the tgw-rt-shared route table. It was correctly associated with tgw-rt-spokes, which is what the onboarding runbook checked — and those are two different things.

Because 10.2.0.0/16 is absent from tgw-rt-shared, the shared services attachment has no route back to spoke-b. The request reaches the service, the service answers, and the reply is discarded at the Transit Gateway.

Association versus propagation#

This is the single biggest conceptual hurdle with Transit Gateway, and it is worth stating plainly because the AWS console presents the two as adjacent tabs with no indication that you need both.

| | Association | Propagation | | --- | --- | --- | | Answers | "Which route table does traffic from this attachment consult?" | "Which route tables learn the CIDRs behind this attachment?" | | Direction | Ingress — how this attachment sends | Advertisement — how others reach this attachment | | Cardinality | Exactly one route table per attachment | Any number of route tables per attachment | | Symptom when missing | The attachment cannot send anywhere | Others cannot reach this attachment |

An attachment can be fully associated and completely unreachable. That is exactly what happened here: spoke-b could send (association present, route to shared present) but could not be answered (propagation absent).

The reason the failure looked so strange is that it is direction-asymmetric at the hub, not at the endpoint. Both spokes had identical egress. Only the return path differed, and the return path is governed by a route table neither spoke is associated with.

Why spoke-a to spoke-b also fails, and why that is fine#

Same mechanism, deliberate outcome. tgw-rt-spokes has only 10.0.0.0/16 propagated into it, so a packet from spoke-a destined for 10.2.x.x finds no match and is dropped. No spoke CIDRs in the spoke route table is precisely how you implement spoke isolation on a Transit Gateway.

Being able to say "this failure is the design working" is part of the job. Not every dropped packet is a bug.

Distinguishing a missing route from a blackhole#

The defect here was an absent row, which is the harder of the two to spot. The other way Transit Gateway routing fails leaves visible evidence:

Practical consequence: when a Transit Gateway path fails, do not just read the route table looking for something wrong. Search it for the specific destination and treat an empty result as a finding in its own right. route-search.longest-prefix-match exists for exactly this.

The fix#

One resource. The propagation that onboarding skipped.

If you are converting an existing deployment, the old resource needs a state move rather than a destroy and recreate:

terraform state mv \
  'aws_ec2_transit_gateway_route_table_propagation.spoke_a_into_shared' \
  'aws_ec2_transit_gateway_route_table_propagation.spokes_into_shared["spoke-a"]'
 
terraform apply
# Plan: 1 to add, 0 to change, 0 to destroy.

Verify#

From spoke-b, the command that failed:

curl -sS -m 8 http://10.0.1.20:8080/
# vn-lab-03 shared ok

Then confirm the route now exists:

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id "$SHARED" \
  --filters "Name=route-search.longest-prefix-match,Values=10.2.1.20" \
  --query 'Routes[].[DestinationCidrBlock,Type,State]' --output table
# 10.2.0.0/16  propagated  active

And confirm the isolation policy still holds — this should still fail:

# From spoke-a
curl -sS -m 5 http://10.2.1.20:8080/
# curl: (28) Connection timed out after 5001 milliseconds

Senior debrief#

The transferable lesson: on a Transit Gateway, reachability is decided by the route table associated with the ingress attachment. Every path therefore involves at least two route table lookups — one per direction — and they can be governed by different tables with different contents. Reasoning about a Transit Gateway path in one direction is how this bug survives.

Why the runbook failed. It had a step for association and no step for propagation, and association is the one the console surfaces most prominently. The runbook was completed correctly and the outcome was still broken, which is the most expensive kind of process defect: it produces confident, incorrect verification.

Why "identical module" was misleading. It was true and irrelevant. Both spoke VPCs were genuinely identical; the difference lived in the hub, in a resource that names the spoke but is not part of the spoke's own module. Any time a topology has per-tenant resources that live outside the tenant's module, that boundary is where onboarding gaps accumulate. Generate them from the same data structure and the gap cannot open.

How this presents in an interview. "One spoke works, one does not, both attachments are associated" is a common Transit Gateway prompt precisely because it separates people who have memorized the words from people who have debugged it. The strong answer distinguishes association from propagation without prompting, then asks which direction fails and which route table governs it. Mentioning that spoke-to-spoke isolation is achieved through the absence of propagated routes demonstrates you understand the mechanism rather than a recipe.

Guardrails worth adding:

• Derive propagations from the same map that creates attachments. The fix above is not just tidier; it removes the failure mode. Any onboarding step a human can skip will eventually be skipped.
• Add a reachability test to onboarding, not a configuration check. Asserting "the attachment is associated" passed here. Asserting "the new spoke can complete a request to shared services and cannot reach another spoke" would not have.
• Alarm on Transit Gateway PacketDropCountNoRoute. This CloudWatch metric increments for exactly this failure and was non-zero from the moment spoke-b was onboarded.
• Review Transit Gateway route tables as policy, not plumbing. What is missing from a route table is as load-bearing as what is in it — spoke isolation here depends entirely on an absence. A reviewer who only reads added lines cannot evaluate that.

Clean up#

terraform destroy

Then confirm, because attachments are the expensive part and a partial destroy is easy to miss:

aws ec2 describe-transit-gateway-attachments \
  --query 'TransitGatewayAttachments[?State!=`deleted`].[TransitGatewayAttachmentId,State]' \
  --output table
# Should be empty.