Lab 11: The Egress That Worked Half The Time
Outbound connections from a private subnet succeed sometimes and time out other times, with no pattern. Retries usually work. The route tables are right, the security groups are open, and the NAT Gateway reports healthy.
- Debugging time
- ~30 min
- Reading time
- 13 min
- Reported by
- Integrations
- Tier
- Professional
Outbound API calls fail intermittently from the workload subnet
Reported by Integrations
Roughly half of our outbound calls to partner APIs time out. The other half work. Same host, same destination, same command — run it twice and you get two different results.
Our retry logic mostly covers it, which is why this took a week to escalate. It is now causing duplicate submissions on a partner endpoint that is not idempotent, so it has to be fixed properly.
The route tables are correct. The security group allows all egress. The NAT Gateway shows no errors in CloudWatch. We tried a larger instance in case it was a connection limit. No change.
The one thing we did recently was tighten the network ACL on the NAT subnet as part of a hardening review. That was three weeks before the symptoms started being reported, so we do not think it is related.
What you are working with
One VPC, one Availability Zone, standard two-tier egress.
Only one of the two subnets has an explicit network ACL, and it is not the one holding the workload. Both subnets are in the same zone, so nothing here is a cross-AZ problem.
| Resource | Configuration |
|---|---|
| VPC | 10.90.0.0/16 |
| NAT subnet | 10.90.1.0/24, route 0.0.0.0/0 → internet gateway |
| Workload subnet | 10.90.11.0/24, route 0.0.0.0/0 → NAT Gateway |
| NAT Gateway | In the NAT subnet, available |
nacl-nat inbound | 100: allow TCP 32768-65535 from 0.0.0.0/0; 110: allow all from 10.90.0.0/16 |
nacl-nat outbound | 100: allow TCP 443; 105: allow TCP 80; 110: allow all to 10.90.0.0/16 |
| Workload subnet NACL | VPC default — allow all, both directions |
sg-workload | All outbound, no inbound |
The workload subnet uses the default permissive NACL. That is worth registering early: whatever is being dropped is not being dropped anywhere near the workload.
- EC2
Workload
10.90.11.5:51234 → partner:443
- FILTER
NAT subnet inbound
rule 110 — allow all from 10.90.0.0/16
- GW
NAT Gateway outbound
rewrites source to EIP:P, where P is 1024-65535
- FILTER
NAT subnet inbound, on the reply
reply arrives at EIP:P — rule 100 admits 32768-65535 only
Dropped — When P is below 32768 no inbound rule matches the reply and it hits the implicit deny. When P is 32768 or above the same connection succeeds. Nothing about the request differs.
- DEST
Workload
never receives the reply, retransmits, eventually times out
The outcome depends on a port number the NAT Gateway chooses per connection, which is why the same command gives different answers.
Scope and constraints
- In scope: why the outcome varies per connection.
- Out of scope: route tables, security groups, the instance, and the NAT Gateway's health. All are correct and the NAT Gateway is genuinely fine.
- This is not port exhaustion.
ErrorPortAllocationstays at zero and the workload makes a handful of connections, not tens of thousands. - This is not Lab 01. The NAT Gateway is in the right subnet with the right route.
- This is not Lab 02 either, although it is the same underlying property of NACLs. There the return path was blocked for every connection. Here it is blocked for some.
Deploy the broken state
cd lab-11-nacl-nat-ephemeral-ports
terraform init
terraform apply
aws ssm start-session --target "$(terraform output -raw workload_instance_id)"Session Manager reaches this host over the NAT Gateway, so the agent's own connection hits the same
defect. It retries and registers within a minute or two. If start-session fails, run it again —
that is the symptom, not a broken lab.
Establish that it is intermittent
The single most important step is to stop testing once and start counting.
for i in $(seq 1 20); do
if curl -s -m 5 https://checkip.amazonaws.com >/dev/null; then
echo "$i ok"
else
echo "$i FAIL"
fi
done1 ok
2 FAIL
3 ok
4 ok
5 FAIL
6 FAIL
7 ok
8 ok
9 FAIL
10 ok
...A mix, in no order. The exact ratio will vary between runs — it depends on which source ports the NAT Gateway happens to assign — so do not read anything into the precise number. What matters is that it is neither zero nor twenty.
Run the successful case once more and note what comes back:
curl -s -m 5 https://checkip.amazonaws.com
# 54.161.44.19That is the NAT Gateway's Elastic IP, which confirms the successful connections really are going out through the NAT Gateway and not by some other path.
Rule out the usual suspects
Port allocation errors, which would be the obvious NAT Gateway explanation:
aws cloudwatch get-metric-statistics --region us-east-1 \
--namespace AWS/NATGateway --metric-name ErrorPortAllocation \
--dimensions Name=NatGatewayId,Value="$(terraform output -raw nat_gateway_id)" \
--start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--period 300 --statistics Sum \
--query 'Datapoints[].Sum'
# []Nothing. And connection counts are trivial:
aws cloudwatch get-metric-statistics --region us-east-1 \
--namespace AWS/NATGateway --metric-name ActiveConnectionCount \
--dimensions Name=NatGatewayId,Value="$(terraform output -raw nat_gateway_id)" \
--start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--period 300 --statistics Maximum \
--query 'Datapoints[].Maximum'
# [ 4.0 ]Four. Whatever this is, it is not capacity.
See where the packet dies
A NACL denial is invisible from both hosts — the sender sees a timeout and the receiver sees nothing. Flow logs on the NAT subnet are the only place the drop is recorded.
Generate some failures, then read the log:
for i in $(seq 1 20); do curl -s -m 5 https://checkip.amazonaws.com >/dev/null; doneaws logs filter-log-events --region us-east-1 \
--log-group-name "$(terraform output -raw flow_log_group)" \
--filter-pattern REJECT \
--query 'events[0:6].message' --output text2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 6153 6 1 52 … REJECT OK
2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 14620 6 1 52 … REJECT OK
2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 28115 6 1 52 … REJECT OK
2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 9877 6 1 52 … REJECT OKRead the fields. Source is the partner on port 443. Destination is the NAT Gateway's ENI, on ports
6153, 14620, 28115, 9877. These are replies being rejected on the way in.
Now the accepted ones, for contrast:
aws logs filter-log-events --region us-east-1 \
--log-group-name "$(terraform output -raw flow_log_group)" \
--filter-pattern "ACCEPT" \
--query 'events[0:6].message' --output text | awk '{print $6, $7, $NF}'443 41288 OK
443 57002 OK
443 39114 OKEvery rejected destination port is below 32768. Every accepted one is above it. That is the pattern, and it is a property of the port number rather than of the host, the destination, or the time.
Read the NACL against that boundary
aws ec2 describe-network-acls --region us-east-1 \
--network-acl-ids "$(terraform output -raw nat_nacl_id)" \
--query 'NetworkAcls[0].Entries[?Egress==`false`].[RuleNumber,Protocol,RuleAction,CidrBlock,PortRange]' \
--output json
# [
# [100, "6", "allow", "0.0.0.0/0", {"From": 32768, "To": 65535}],
# [110, "-1", "allow", "10.90.0.0/16", null],
# [32767, "-1", "deny", "0.0.0.0/0", null]
# ]32768 appears in the NACL and 32768 is the boundary in the flow logs. Confirm the NACL is on the NAT
subnet rather than the workload subnet:
terraform output nat_subnet_id
aws ec2 describe-network-acls --region us-east-1 \
--network-acl-ids "$(terraform output -raw nat_nacl_id)" \
--query 'NetworkAcls[0].Associations[].SubnetId' --output textThis debrief is part of Labs Pro
The root-cause analysis, packet-flow walkthrough, and Terraform remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The brief, the reproduction steps, and the Terraform stay free — you can still solve this one yourself.
Already bought it? Sign in and it unlocks.