Skip to content
VenerableNetworks
ProfessionalProAWS VPCNAT Gateway

Lab 16: The Borrowed NAT Gateway

Centralised egress across a peering connection. The default route is active, the peering is active, and the NAT Gateway is provably healthy because another instance is using it right now. Nothing in the application VPC can reach the internet.

Debugging time
~25 min
Reading time
11 min
Reported by
Platform Engineering
Tier
Professional
INC-1704SEV-2InvestigatingOpened 2026-09-26 09:20 UTC

Application VPC has no outbound internet after moving to shared egress

Reported by Platform Engineering

We consolidated NAT. Instead of a NAT Gateway per VPC we now have one egress VPC, peered to everything, and each VPC routes 0.0.0.0/0 at the peering connection.

The egress VPC works fine. Its own private instances reach the internet through the NAT Gateway right now — I checked before opening this.

The application VPC reaches nothing. The peering connection is active. The default route is in the route table and the console shows it as active, not blackhole. The NAT Gateway is available. Security groups allow all egress.

The application VPC can reach the egress VPC over the peering without any problem, so the peering is definitely carrying traffic. It just will not carry this traffic.

What you are working with

Two VPCs in one Availability Zone. No Transit Gateway.

| Resource | Configuration | | --- | --- | | Egress VPC | 10.150.0.0/16, internet gateway, NAT Gateway, private subnet | | Application VPC | 10.151.0.0/16, no internet gateway, no NAT Gateway | | Peering | active | | App route table | 10.150.0.0/16 → pcx, 0.0.0.0/0 → pcx | | Egress private route table | 0.0.0.0/0 → NAT Gateway, 10.151.0.0/16 → pcx | | Control host | Egress VPC private subnet, reaches the internet through the NAT Gateway | | App host | Application VPC, reaches nothing |

The application host also has Systems Manager interface endpoints, which is the only reason you can get a shell on it. They resolve inside the VPC, so they work regardless of the egress problem.

  1. EC2

    App host

    10.151.1.x → a public address

  2. RTB

    App route table

    0.0.0.0/0 → pcx, State: active

  3. GW

    Peering connection

    active, and carrying VPC-to-VPC traffic correctly

    Dropped — Peering does not support edge-to-edge routing. It will carry traffic addressed to the peer VPC and will not carry traffic onward to that VPC’s NAT Gateway, internet gateway, VPN, or Direct Connect link.

  4. GW

    NAT Gateway

    available, healthy, in use by another instance — never reached

Every object on this path is healthy and the route reports active. The packet is discarded at the peering connection because of what peering is, not because anything is misconfigured.

Scope and constraints

  • In scope: why the peering connection carries VPC-to-VPC traffic and not internet-bound traffic.
  • Out of scope: the NAT Gateway, the peering connection's state, the route table's contents, and the security groups. All are healthy, and two of them are provably healthy in one command each.
  • The default route really is active. This is not a blackhole route and not a pending-acceptance peering.
  • The NAT Gateway really is working. Another instance is using it while you debug.
  • Adding a second route, reordering, or making the route more specific does not fix this.

Deploy the broken state

cd lab-16-peering-borrowed-nat
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw app_instance_id)"

The NAT Gateway takes about ninety seconds and the interface endpoints about thirty. Session Manager will not connect to the application host until the endpoints are available.