Lab 09: The Reply That Went To The Wrong VPC

What you are working with#

A hub VPC peered to two team VPCs. One Availability Zone throughout.

| Resource | Configuration | | --- | --- | | Hub VPC | 10.70.0.0/16, service at 10.70.1.50:5000 | | Team B VPC | 10.71.0.0/16, client at 10.71.1.100 | | Team C VPC | 10.71.0.0/16, host at 10.71.1.100 | | pcx-hub-b | active | | pcx-hub-c | active | | Hub route table | 10.71.0.0/16 → pcx-hub-c | | Team B route table | 10.70.0.0/16 → pcx-hub-b | | Team C route table | 10.70.0.0/16 → pcx-hub-c |

Read that table again before going further. Everything you need is in it, and most people look straight past it because each row is individually correct.

Scope and constraints#

• In scope: the hub's return path.
• Out of scope: team B's route table, both peering connections, and all three security groups. Every one of them is correct, and the lab is partly about proving that fast rather than assuming it.
• Both peerings are active. This is not a pending-acceptance blackhole.
• The service is listening and answering. The hub team is right about that.
• The fix is not a route change in the hub route table. Convince yourself of that before reading the solution — it is the most valuable thing this lab has to teach.

Deploy the broken state#

cd lab-09-peering-reverse-path
terraform init
terraform apply
 
# Shell on team B's client
aws ssm start-session --target "$(terraform output -raw team_b_instance_id)"

You will want three sessions open for this one: team B's client, the hub service, and team C's host. Give the apply a minute or two for tcpdump to install on all three.

SVC=$(terraform output -raw hub_service_ip)     # 10.70.1.50
PORT=$(terraform output -raw service_port)      # 5000

Confirm the failure#

From team B's client:

nc -vz -w 5 "$SVC" "$PORT"
# Ncat: Version 7.93 ( https://nmap.org/ncat )
# Ncat: Connection timed out.

A timeout. Per the triage table, that means a packet left and nothing came back — routing, a NACL, or a security group. Hold that thought, because this lab is about to violate the useful half of it.

Confirm it works from team C#

From team C's host, the identical command:

nc -w 3 "$SVC" "$PORT"
# lab-09-shared-service-ok

Same service, same port, same CIDR, same module. One works.

Watch the request arrive at the hub#

On the hub service host, start a capture, then re-run the failing connection from team B.

# On the hub
sudo tcpdump -ni any "tcp port 5000" -c 6

# On team B, while the capture runs
nc -vz -w 5 "$SVC" "$PORT"

The hub sees this:

14:22:03.118 IP 10.71.1.100.44310 > 10.70.1.50.5000: Flags [S], seq 3419927154
14:22:03.118 IP 10.70.1.50.5000 > 10.71.1.100.44310: Flags [S.], seq 1882337401, ack 3419927155
14:22:04.121 IP 10.71.1.100.44310 > 10.70.1.50.5000: Flags [S], seq 3419927154
14:22:04.121 IP 10.70.1.50.5000 > 10.71.1.100.44310: Flags [S.], seq 1882337401, ack 3419927155

This is the finding that breaks the normal checklist. The SYN arrives. The hub answers with a SYN-ACK. The client retransmits, because it never received that SYN-ACK, and the hub answers again.

The hub is doing everything right. The reply is leaving. It is simply not arriving at team B.

Find where the reply went#

This is the step that most people never think to take, and it is the whole lab.

On team C's host — a VPC with no involvement in this connection whatsoever — start a capture:

# On team C
sudo tcpdump -ni any "tcp port 5000" -c 4

Re-run the failing connection from team B once more, and watch team C's capture:

14:24:41.907 IP 10.70.1.50.5000 > 10.71.1.100.44502: Flags [S.], seq 2298104411, ack 1174559832
14:24:41.907 IP 10.71.1.100.44502 > 10.70.1.50.5000: Flags [R], seq 1174559832
14:24:42.911 IP 10.70.1.50.5000 > 10.71.1.100.44502: Flags [S.], seq 2298104411, ack 1174559832
14:24:42.911 IP 10.71.1.100.44502 > 10.70.1.50.5000: Flags [R], seq 1174559832

Team C's host is receiving SYN-ACKs for a connection it never opened, and its kernel is correctly answering each one with a RST. Team B's replies are being delivered to team C.

Note the source port — 44502 here — and compare it to the port team B's client used on the same attempt. They match. This is not similar traffic. It is the same TCP connection.

Read the hub route table#

aws ec2 describe-route-tables \
  --route-table-ids "$(terraform output -raw hub_route_table_id)" \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,VpcPeeringConnectionId,GatewayId,State]' \
  --output table
# --------------------------------------------------------------
# |  10.70.0.0/16  |  None            |  local      |  active  |
# |  10.71.0.0/16  |  pcx-0c5d…       |  None       |  active  |
# |  0.0.0.0/0     |  None            |  igw-0a1b…  |  active  |
# --------------------------------------------------------------

One route for 10.71.0.0/16, pointing at one peering connection. Confirm which:

terraform output pcx_hub_b
terraform output pcx_hub_c

The route names pcx-hub-c. Every reply the hub sends to 10.71.0.0/16 leaves over team C's peering, whichever team asked.