Skip to content
VenerableNetworks
ProfessionalProAWS VPCNAT Gateway

Lab 16: The Borrowed NAT Gateway

Centralised egress across a peering connection. The default route is active, the peering is active, and the NAT Gateway is provably healthy because another instance is using it right now. Nothing in the application VPC can reach the internet.

Debugging time
~25 min
Reading time
12 min
Reported by
Platform Engineering
Tier
Professional
INC-1704SEV-2InvestigatingOpened 2026-09-26 09:20 UTC

Application VPC has no outbound internet after moving to shared egress

Reported by Platform Engineering

We consolidated NAT. Instead of a NAT Gateway per VPC we now have one egress VPC, peered to everything, and each VPC routes 0.0.0.0/0 at the peering connection.

The egress VPC works fine. Its own private instances reach the internet through the NAT Gateway right now — I checked before opening this.

The application VPC reaches nothing. The peering connection is active. The default route is in the route table and the console shows it as active, not blackhole. The NAT Gateway is available. Security groups allow all egress.

The application VPC can reach the egress VPC over the peering without any problem, so the peering is definitely carrying traffic. It just will not carry this traffic.

What you are working with

Two VPCs in one Availability Zone. No Transit Gateway.

Starting state — two VPCs, one internet gateway
Application VPC · 10.151.0.0/16 Availability Zone A contains app subnet 10.151.1.0/24, default route 0.0.0.0/0 → pcx, holding App host, SSM endpoints. Egress VPC · 10.150.0.0/16 Attached: Internet gateway. Availability Zone A contains egress public 10.150.1.0/24, default route 0.0.0.0/0, holding NAT Gateway; egress private 10.150.11.0/24, default route 0.0.0.0/0, holding Control host.the internetVPC peeringactiveApplication VPC · 10.151.0.0/16Availability Zone Aapp subnetprivate10.151.1.0/240.0.0.0/0 → pcxApp hostno outboundpathSSMendpointsprivate DNSonEgress VPC · 10.150.0.0/16Availability Zone Aegress publicpublic10.150.1.0/24NAT Gatewayavailable, EIP attachedegress privateprivate10.150.11.0/24Control hostreaches the internet0.0.0.0/00.0.0.0/0Internet gateway

The application VPC has no internet gateway and no NAT Gateway of its own. Its default route points at the peering connection, which is active. The control host in the egress VPC reaches the internet through the NAT Gateway; the application host is expected to borrow it.

ResourceConfiguration
Egress VPC10.150.0.0/16, internet gateway, NAT Gateway, private subnet
Application VPC10.151.0.0/16, no internet gateway, no NAT Gateway
Peeringactive
App route table10.150.0.0/16 → pcx, 0.0.0.0/0 → pcx
Egress private route table0.0.0.0/0 → NAT Gateway, 10.151.0.0/16 → pcx
Control hostEgress VPC private subnet, reaches the internet through the NAT Gateway
App hostApplication VPC, reaches nothing

The application host also has Systems Manager interface endpoints, which is the only reason you can get a shell on it. They resolve inside the VPC, so they work regardless of the egress problem.

  1. EC2

    App host

    10.151.1.x → a public address

  2. RTB

    App route table

    0.0.0.0/0 → pcx, State: active

  3. GW

    Peering connection

    active, and carrying VPC-to-VPC traffic correctly

    Dropped — Peering does not support edge-to-edge routing. It will carry traffic addressed to the peer VPC and will not carry traffic onward to that VPC’s NAT Gateway, internet gateway, VPN, or Direct Connect link.

  4. GW

    NAT Gateway

    available, healthy, in use by another instance — never reached

Every object on this path is healthy and the route reports active. The packet is discarded at the peering connection because of what peering is, not because anything is misconfigured.

Scope and constraints

  • In scope: why the peering connection carries VPC-to-VPC traffic and not internet-bound traffic.
  • Out of scope: the NAT Gateway, the peering connection's state, the route table's contents, and the security groups. All are healthy, and two of them are provably healthy in one command each.
  • The default route really is active. This is not a blackhole route and not a pending-acceptance peering.
  • The NAT Gateway really is working. Another instance is using it while you debug.
  • Adding a second route, reordering, or making the route more specific does not fix this.

Deploy the broken state

cd lab-16-peering-borrowed-nat
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw app_instance_id)"

The NAT Gateway takes about ninety seconds and the interface endpoints about thirty. Session Manager will not connect to the application host until the endpoints are available.