Lab 11: The Egress That Worked Half The Time

What you are working with#

One VPC, one Availability Zone, standard two-tier egress.

| Resource | Configuration | | --- | --- | | VPC | 10.90.0.0/16 | | NAT subnet | 10.90.1.0/24, route 0.0.0.0/0 → internet gateway | | Workload subnet | 10.90.11.0/24, route 0.0.0.0/0 → NAT Gateway | | NAT Gateway | In the NAT subnet, available | | nacl-nat inbound | 100: allow TCP 32768-65535 from 0.0.0.0/0; 110: allow all from 10.90.0.0/16 | | nacl-nat outbound | 100: allow TCP 443; 105: allow TCP 80; 110: allow all to 10.90.0.0/16 | | Workload subnet NACL | VPC default — allow all, both directions | | sg-workload | All outbound, no inbound |

The workload subnet uses the default permissive NACL. That is worth registering early: whatever is being dropped is not being dropped anywhere near the workload.

Scope and constraints#

• In scope: why the outcome varies per connection.
• Out of scope: route tables, security groups, the instance, and the NAT Gateway's health. All are correct and the NAT Gateway is genuinely fine.
• This is not port exhaustion. ErrorPortAllocation stays at zero and the workload makes a handful of connections, not tens of thousands.
• This is not Lab 01. The NAT Gateway is in the right subnet with the right route.
• This is not Lab 02 either, although it is the same underlying property of NACLs. There the return path was blocked for every connection. Here it is blocked for some.

Deploy the broken state#

cd lab-11-nacl-nat-ephemeral-ports
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw workload_instance_id)"

Session Manager reaches this host over the NAT Gateway, so the agent's own connection hits the same defect. It retries and registers within a minute or two. If start-session fails, run it again — that is the symptom, not a broken lab.

Establish that it is intermittent#

The single most important step is to stop testing once and start counting.

for i in $(seq 1 20); do
  if curl -s -m 5 https://checkip.amazonaws.com >/dev/null; then
    echo "$i ok"
  else
    echo "$i FAIL"
  fi
done

1 ok
2 FAIL
3 ok
4 ok
5 FAIL
6 FAIL
7 ok
8 ok
9 FAIL
10 ok
...

A mix, in no order. The exact ratio will vary between runs — it depends on which source ports the NAT Gateway happens to assign — so do not read anything into the precise number. What matters is that it is neither zero nor twenty.

Run the successful case once more and note what comes back:

curl -s -m 5 https://checkip.amazonaws.com
# 54.161.44.19

That is the NAT Gateway's Elastic IP, which confirms the successful connections really are going out through the NAT Gateway and not by some other path.

Rule out the usual suspects#

Port allocation errors, which would be the obvious NAT Gateway explanation:

aws cloudwatch get-metric-statistics --region us-east-1 \
  --namespace AWS/NATGateway --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value="$(terraform output -raw nat_gateway_id)" \
  --start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 300 --statistics Sum \
  --query 'Datapoints[].Sum'
# []

Nothing. And connection counts are trivial:

aws cloudwatch get-metric-statistics --region us-east-1 \
  --namespace AWS/NATGateway --metric-name ActiveConnectionCount \
  --dimensions Name=NatGatewayId,Value="$(terraform output -raw nat_gateway_id)" \
  --start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 300 --statistics Maximum \
  --query 'Datapoints[].Maximum'
# [ 4.0 ]

Four. Whatever this is, it is not capacity.

See where the packet dies#

A NACL denial is invisible from both hosts — the sender sees a timeout and the receiver sees nothing. Flow logs on the NAT subnet are the only place the drop is recorded.

Generate some failures, then read the log:

for i in $(seq 1 20); do curl -s -m 5 https://checkip.amazonaws.com >/dev/null; done

aws logs filter-log-events --region us-east-1 \
  --log-group-name "$(terraform output -raw flow_log_group)" \
  --filter-pattern REJECT \
  --query 'events[0:6].message' --output text

2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 6153  6 1 52 … REJECT OK
2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 14620 6 1 52 … REJECT OK
2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 28115 6 1 52 … REJECT OK
2 1234… eni-0ab… 52.94.236.248 10.90.1.204 443 9877  6 1 52 … REJECT OK

Read the fields. Source is the partner on port 443. Destination is the NAT Gateway's ENI, on ports 6153, 14620, 28115, 9877. These are replies being rejected on the way in.

Now the accepted ones, for contrast:

aws logs filter-log-events --region us-east-1 \
  --log-group-name "$(terraform output -raw flow_log_group)" \
  --filter-pattern "ACCEPT" \
  --query 'events[0:6].message' --output text | awk '{print $6, $7, $NF}'

443 41288 OK
443 57002 OK
443 39114 OK

Every rejected destination port is below 32768. Every accepted one is above it. That is the pattern, and it is a property of the port number rather than of the host, the destination, or the time.

Read the NACL against that boundary#

aws ec2 describe-network-acls --region us-east-1 \
  --network-acl-ids "$(terraform output -raw nat_nacl_id)" \
  --query 'NetworkAcls[0].Entries[?Egress==`false`].[RuleNumber,Protocol,RuleAction,CidrBlock,PortRange]' \
  --output json
# [
#   [100, "6", "allow", "0.0.0.0/0", {"From": 32768, "To": 65535}],
#   [110, "-1", "allow", "10.90.0.0/16", null],
#   [32767, "-1", "deny", "0.0.0.0/0", null]
# ]

32768 appears in the NACL and 32768 is the boundary in the flow logs. Confirm the NACL is on the NAT subnet rather than the workload subnet:

terraform output nat_subnet_id
aws ec2 describe-network-acls --region us-east-1 \
  --network-acl-ids "$(terraform output -raw nat_nacl_id)" \
  --query 'NetworkAcls[0].Associations[].SubnetId' --output text