Skip to content
VenerableNetworks
ProfessionalFree accountTransit GatewayAWS VPC

Lab 03: The Spoke That Could Not Be Answered

Two spoke VPCs built from the same Terraform module. One reaches shared services, the other times out. Every attachment is available, both VPC route tables are identical, and the Transit Gateway shows both spokes correctly associated.

Debugging time
~30 min
Reading time
13 min
Reported by
Platform Networking
Tier
Professional
INC-1142SEV-2InvestigatingOpened Today 10:47 UTC

Newly onboarded spoke VPC cannot reach shared services; the existing spoke works

Reported by Platform Networking

Environment
staging
Region
us-east-1
Working spoke
spoke-a — 10.1.0.0/16
Broken spoke
spoke-b — 10.2.0.0/16
Target
shared services — 10.0.0.0/16

We onboarded spoke-b yesterday using the same Terraform module that built spoke-a. The internal config API in shared services is reachable from spoke-a and times out from spoke-b.

The two VPCs are as close to identical as two VPCs get:

  • Same module, same version, only the CIDR inputs differ.
  • Both VPC route tables send 10.0.0.0/8 at the Transit Gateway. We diffed them.
  • Both Transit Gateway attachments report available.
  • Both attachments are associated with the tgw-rt-spokes route table. We checked this specifically because it is the usual culprit.
  • Security groups on the shared services host allow the service port from 10.0.0.0/8, which covers both spokes.

The onboarding runbook has three steps and we completed all three. Networking has looked at it for two hours. We are not sure what is left to check.

What you are working with

A standard hub-and-spoke Transit Gateway topology. Two Transit Gateway route tables implement the policy: spokes may reach shared services, spokes may not reach each other.

Starting state — the VPC layer is identical in all three networks
VPC spoke-a 10.1.0.0/16 AZ a contains spoke-a 10.1.1.0/24, default route 10.0.0.0/8 → tgw, holding Spoke host. VPC shared services 10.0.0.0/16 AZ a contains shared 10.0.1.0/24, default route 10.0.0.0/8 → tgw, holding Shared service. VPC spoke-b 10.2.0.0/16 AZ a contains spoke-b 10.2.1.0/24, default route 10.0.0.0/8 → tgw, holding Spoke host.transitgatewayattachmentassociatedtransitgatewayattachmentassociatedVPC spoke-a 10.1.0.0/16AZ aspoke-apublic10.1.1.0/2410.0.0.0/8 → tgwSpoke hostlistens on 8080VPC shared services 10.0.0.0/16AZ asharedpublic10.0.1.0/2410.0.0.0/8 → tgwShared servicelistens on 8080VPC spoke-b 10.2.0.0/16AZ aspoke-bpublic10.2.1.0/2410.0.0.0/8 → tgwSpoke hostlistens on 8080

There is one transit gateway; it is drawn on each path it carries. Every VPC is built from the same definition — same summary route at 10.0.0.0/8, same attachment, same association — and each also holds a 0.0.0.0/0 route to its own internet gateway so the hosts can install tooling. Nothing at this layer distinguishes the spoke that works from the one that does not, which is why the answer is not in this picture.

10.2.1.20:52104 → 10.0.1.20:8080 (TCP SYN, spoke-b to shared)
  1. EC2

    spoke-b host

    10.2.1.20 in vpc spoke-b (10.2.0.0/16)

  2. RTB

    spoke-b VPC route table

    10.0.0.0/8 → tgw-…

  3. RTB

    TGW route table: tgw-rt-spokes

    consulted because the spoke-b attachment is associated here

  4. GW

    shared services attachment

    10.0.0.0/16 is propagated into tgw-rt-spokes

  5. DEST

    shared services host

    10.0.1.20:8080 — listening

The request path is complete and correct. As in Lab 02, the interesting direction is the one nobody drew.

Scope and constraints

  • In scope: Transit Gateway route tables, associations, and propagations.
  • Out of scope: security groups, NACLs, VPC route tables, DNS, and the service itself. All are correct and identical across both spokes.
  • spoke-a cannot reach spoke-b, and that is intentional. Spoke-to-spoke isolation is the point of this topology. Do not chase it as a second bug — but do be able to explain why it fails, because it is the same mechanism as the real defect.
  • You get a shell on all three hosts. Every host runs the same listener on port 8080, so you can probe in any direction.

Deploy the broken state

main.tf8 lines

Download the file below into an empty directory and apply it. Expect four to six minutes — Transit Gateway attachments are the slow part.

terraform init
terraform apply

# Shells for all three hosts
terraform output start_session_commands

# Private IPs to probe
terraform output private_ips

Full source: main.tf. Three VPCs generated from one for_each, so the VPC layer is provably symmetric, plus the Transit Gateway, two route tables, and the association and propagation resources.