Skip to content
VenerableNetworks
SpecialtyProAWS VPC

Lab 09: The Reply That Went To The Wrong VPC

Two teams peer to the same hub with the same VPC module and the same CIDR. One reaches the shared service, the other times out. The requests arrive at the hub. The replies are delivered to the other team.

Debugging time
~35 min
Reading time
13 min
Reported by
Shared Platform
Tier
Specialty
INC-1478SEV-2InvestigatingOpened 2026-09-18 14:05 UTC

Team B cannot reach the shared directory service; team C can

Reported by Shared Platform

Team B onboarded to the shared hub this morning using the same runbook team C used last week. Their peering connection is active. Their route to the hub is in place. The hub's security group already permits their CIDR on the service port.

Every connection from team B to the service times out. Team C, on the identical configuration, works.

The hub team says they can see team B's requests arriving — they ran tcpdump on the service host and the packets are there. The service is answering them. Team B says nothing ever comes back.

Both teams used our VPC module, so their networks are configured the same way. We are stuck on how a request can arrive and be answered and still time out.

What you are working with

A hub VPC peered to two team VPCs. One Availability Zone throughout.

Starting state — three VPCs, two peerings, and the addresses each one uses
Team B VPC · 10.71.0.0/16 Attached: Internet gateway. AZ a contains team-b subnet 10.71.1.0/24, default route 10.70.0.0/16 → pcx-b, holding Client. Hub VPC · 10.70.0.0/16 Attached: Internet gateway. AZ a contains hub subnet 10.70.1.0/24, default route 10.71.0.0/16 → pcx-c, holding Service. Team C VPC · 10.71.0.0/16 Attached: Internet gateway. AZ a contains team-c subnet 10.71.1.0/24, default route 10.70.0.0/16 → pcx-c, holding Client.pcx-bactivepcx-cactiveTeam B VPC · 10.71.0.0/16AZ ateam-b subnetpublic10.71.1.0/2410.70.0.0/16 → pcx-bClient10.71.1.100Hub VPC · 10.70.0.0/16AZ ahub subnetpublic10.70.1.0/2410.71.0.0/16 → pcx-cService10.70.1.50:5000Team C VPC · 10.71.0.0/16AZ ateam-c subnetpublic10.71.1.0/2410.70.0.0/16 → pcx-cClient10.71.1.100Internet gatewayInternet gatewayInternet gateway

Read the CIDRs on the two team VPCs, then read the single route the hub holds for that prefix. Both teams were cut from the same module and never re-addressed, down to the host address.

ResourceConfiguration
Hub VPC10.70.0.0/16, service at 10.70.1.50:5000
Team B VPC10.71.0.0/16, client at 10.71.1.100
Team C VPC10.71.0.0/16, host at 10.71.1.100
pcx-hub-bactive
pcx-hub-cactive
Hub route table10.71.0.0/16 → pcx-hub-c
Team B route table10.70.0.0/16 → pcx-hub-b
Team C route table10.70.0.0/16 → pcx-hub-c

Read that table again before going further. Everything you need is in it, and most people look straight past it because each row is individually correct.

  1. EC2

    Team B client

    10.71.1.100 — SYN to 10.70.1.50:5000

  2. GW

    pcx-hub-b

    team B route table sends 10.70.0.0/16 here

  3. DEST

    Hub service

    receives the SYN, replies SYN-ACK to 10.71.1.100

  4. RTB

    Hub route table

    10.71.0.0/16 → pcx-hub-c

    Dropped — The reply is routed by destination address alone. Both teams share that CIDR and only one route can exist for it, so the reply leaves over the peering to team C — not the one the request arrived on.

  5. VPCE

    Team C host

    10.71.1.100 — receives a SYN-ACK it never asked for

The forward path is flawless. The failure is entirely in the return path, and the packet is not dropped — it is delivered to the wrong network.

Scope and constraints

  • In scope: the hub's return path.
  • Out of scope: team B's route table, both peering connections, and all three security groups. Every one of them is correct, and the lab is partly about proving that fast rather than assuming it.
  • Both peerings are active. This is not a pending-acceptance blackhole.
  • The service is listening and answering. The hub team is right about that.
  • The fix is not a route change in the hub route table. Convince yourself of that before reading the solution — it is the most valuable thing this lab has to teach.

Deploy the broken state

cd lab-09-peering-reverse-path
terraform init
terraform apply
 
# Shell on team B's client
aws ssm start-session --target "$(terraform output -raw team_b_instance_id)"

You will want three sessions open for this one: team B's client, the hub service, and team C's host. Give the apply a minute or two for tcpdump to install on all three.