Skip to content
VenerableNetworks
ProfessionalProNACL / Security GroupsNAT Gateway

Lab 11: The Egress That Worked Half The Time

Outbound connections from a private subnet succeed sometimes and time out other times, with no pattern. Retries usually work. The route tables are right, the security groups are open, and the NAT Gateway reports healthy.

Debugging time
~30 min
Reading time
13 min
Reported by
Integrations
Tier
Professional
INC-1533SEV-2InvestigatingOpened 2026-09-20 11:18 UTC

Outbound API calls fail intermittently from the workload subnet

Reported by Integrations

Roughly half of our outbound calls to partner APIs time out. The other half work. Same host, same destination, same command — run it twice and you get two different results.

Our retry logic mostly covers it, which is why this took a week to escalate. It is now causing duplicate submissions on a partner endpoint that is not idempotent, so it has to be fixed properly.

The route tables are correct. The security group allows all egress. The NAT Gateway shows no errors in CloudWatch. We tried a larger instance in case it was a connection limit. No change.

The one thing we did recently was tighten the network ACL on the NAT subnet as part of a hardening review. That was three weeks before the symptoms started being reported, so we do not think it is related.

What you are working with

One VPC, one Availability Zone, standard two-tier egress.

Starting state — where the filtering sits
VPC 10.90.0.0/16 Attached: Internet gateway. Availability Zone A contains nat subnet 10.90.1.0/24, default route 0.0.0.0/0, holding NAT Gateway, nacl-nat; workload subnet 10.90.11.0/24, default route 0.0.0.0/0, holding Workload, default NACL.partner APItcp/443VPC 10.90.0.0/16Availability Zone Anat subnetpublic10.90.1.0/24NAT Gatewayavailable, EIP attachednacl-natexplicit rules, both directionsworkload subnetprivate10.90.11.0/24Workloadsg-workload, all outbounddefault NACLallow all both ways0.0.0.0/00.0.0.0/0Internet gateway

Only one of the two subnets has an explicit network ACL, and it is not the one holding the workload. Both subnets are in the same zone, so nothing here is a cross-AZ problem.

ResourceConfiguration
VPC10.90.0.0/16
NAT subnet10.90.1.0/24, route 0.0.0.0/0 → internet gateway
Workload subnet10.90.11.0/24, route 0.0.0.0/0 → NAT Gateway
NAT GatewayIn the NAT subnet, available
nacl-nat inbound100: allow TCP 32768-65535 from 0.0.0.0/0; 110: allow all from 10.90.0.0/16
nacl-nat outbound100: allow TCP 443; 105: allow TCP 80; 110: allow all to 10.90.0.0/16
Workload subnet NACLVPC default — allow all, both directions
sg-workloadAll outbound, no inbound

The workload subnet uses the default permissive NACL. That is worth registering early: whatever is being dropped is not being dropped anywhere near the workload.

  1. EC2

    Workload

    10.90.11.5:51234 → partner:443

  2. FILTER

    NAT subnet inbound

    rule 110 — allow all from 10.90.0.0/16

  3. GW

    NAT Gateway outbound

    rewrites source to EIP:P, where P is 1024-65535

  4. FILTER

    NAT subnet inbound, on the reply

    reply arrives at EIP:P — rule 100 admits 32768-65535 only

    Dropped — When P is below 32768 no inbound rule matches the reply and it hits the implicit deny. When P is 32768 or above the same connection succeeds. Nothing about the request differs.

  5. DEST

    Workload

    never receives the reply, retransmits, eventually times out

The outcome depends on a port number the NAT Gateway chooses per connection, which is why the same command gives different answers.

Scope and constraints

  • In scope: why the outcome varies per connection.
  • Out of scope: route tables, security groups, the instance, and the NAT Gateway's health. All are correct and the NAT Gateway is genuinely fine.
  • This is not port exhaustion. ErrorPortAllocation stays at zero and the workload makes a handful of connections, not tens of thousands.
  • This is not Lab 01. The NAT Gateway is in the right subnet with the right route.
  • This is not Lab 02 either, although it is the same underlying property of NACLs. There the return path was blocked for every connection. Here it is blocked for some.

Deploy the broken state

cd lab-11-nacl-nat-ephemeral-ports
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw workload_instance_id)"

Session Manager reaches this host over the NAT Gateway, so the agent's own connection hits the same defect. It retries and registers within a minute or two. If start-session fails, run it again — that is the symptom, not a broken lab.