Lab 01: The Misplaced NAT Gateway

What you are working with#

A single-AZ VPC with the usual two-tier layout. One public subnet, one private subnet, an internet gateway, and a NAT Gateway that is supposed to give the private tier outbound access.

Scope and constraints#

• In scope: VPC routing, subnet placement, NAT Gateway configuration.
• Out of scope: the third-party API itself, TLS, application code, DNS. The endpoint is reachable from anywhere with working egress.
• The instance has no public IP and must stay that way. Assigning one is not the fix.
• You have Session Manager access to the instance. It works because the lab provisions interface endpoints for SSM inside the VPC, so your shell does not depend on the broken egress path.
• tcpdump and dig are already installed. Neither ships in the AL2023 base image, and an instance with no egress cannot fetch them — so the lab installs both at launch over an S3 gateway endpoint, which is where AL2023 serves its package repositories from. That endpoint carries S3 traffic only, so the egress failure you are here to diagnose is untouched.

Deploy the broken state#

terraform init
terraform apply

# Grab the session command from the outputs
terraform output -raw start_session_command

Full source: main.tf — one file, no modules, no remote state. It provisions a VPC, both subnets, an internet gateway, a NAT Gateway with an Elastic IP, both route tables, SSM interface endpoints, and a t3.micro running Amazon Linux 2023.

Apply takes about three minutes, most of it waiting on the NAT Gateway and the endpoints.

Open a shell on the private instance. Session Manager reaches it through the VPC endpoints, so this works even though outbound internet is broken:

aws ssm start-session --region us-east-1 --target "$(terraform output -raw instance_id)"

Confirm the failure#

Once you have a prompt, try to reach anything on the public internet. Use --max-time so you are not waiting on the default two-minute timeout:

curl -sS --max-time 10 https://checkip.amazonaws.com
# curl: (28) Connection timed out after 10001 milliseconds

The connection hangs and then times out. Note the failure mode: a timeout, not a refusal. That distinction matters and it narrows the problem considerably.

# Refused would mean something actively rejected the packet.
# Timed out means nobody answered at all.
curl -sS --max-time 10 -o /dev/null -w '%{http_code}\n' http://example.com
# curl: (28) Connection timed out after 10002 milliseconds

Rule out DNS#

DNS is often blamed first. Rule it out explicitly — resolution succeeds, so name resolution is not the blocker:

dig +short checkip.amazonaws.com
# 54.235.98.220
# 3.229.87.221
 
# Connect straight to an IP, no DNS involved. Same timeout.
curl -sS --max-time 10 https://54.235.98.220 -k
# curl: (28) Connection timed out after 10003 milliseconds

Watch the packets leave#

Confirm the instance is genuinely transmitting and getting nothing back. Retransmissions with no reply mean the packet is leaving the host and dying somewhere upstream:

sudo tcpdump -ni any 'tcp port 443 and host 54.235.98.220' &
curl -sS --max-time 8 https://54.235.98.220 -k
 
# 09:41:02.118 IP 10.0.11.34.49820 > 54.235.98.220.443: Flags [S], seq 1583...
# 09:41:03.121 IP 10.0.11.34.49820 > 54.235.98.220.443: Flags [S], seq 1583...
# 09:41:05.129 IP 10.0.11.34.49820 > 54.235.98.220.443: Flags [S], seq 1583...

Three SYNs, zero SYN-ACKs. The instance is doing its job. Something between the instance and the internet is swallowing the traffic.

Inspect the path from the outside#

Back on your workstation, walk the routing configuration. Start where the team started — the private route table:

aws ec2 describe-route-tables \
  --route-table-ids "$(terraform output -raw private_route_table_id)" \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,NatGatewayId,GatewayId,State]' \
  --output table

The default route is present and active, pointing at the NAT Gateway. The ticket was accurate. Now ask the question nobody on the ticket asked:

# Which subnet is the NAT Gateway actually in?
aws ec2 describe-nat-gateways \
  --nat-gateway-ids "$(terraform output -raw nat_gateway_id)" \
  --query 'NatGateways[0].[State,SubnetId]' --output text
 
# And which subnets are which?
echo "public:  $(terraform output -raw public_subnet_id)"
echo "private: $(terraform output -raw private_subnet_id)"

Compare those three values carefully. That comparison is the whole lab.

Root cause#

The NAT Gateway was created in the private subnet — the same subnet as the instances it is meant to serve.

aws ec2 describe-nat-gateways --nat-gateway-ids nat-0f8e… \
  --query 'NatGateways[0].SubnetId' --output text
# subnet-0b91…   ← this is subnet-private-a, not public-a

A NAT Gateway is not a router that magically reaches the internet. It is a managed appliance that lives in a subnet and forwards traffic according to that subnet's route table.

The Elastic IP is the part that misleads people, because the NAT Gateway is not what applies it. AWS documents the translation as two steps: both public and private NAT Gateways rewrite the instance's source address to the NAT Gateway's own private address, and for a public NAT Gateway the internet gateway then maps that private address to the Elastic IP. The same page is explicit that a public NAT Gateway only uses its Elastic IP as the source address when it is working alongside an internet gateway in the same VPC.

That is the whole defect in one sentence. The translated packet leaves the NAT Gateway sourced from 10.0.11.x, needs an internet gateway to become the Elastic IP, and the route table it consults — its own subnet's — has no path to one. So the Elastic IP is never applied and the packet has nowhere to go. It is discarded silently, which is why the client sees a timeout rather than an error.

You will see this described elsewhere as a routing loop, on the reasoning that 0.0.0.0/0 points the NAT Gateway back at itself. That is a plausible inference, but AWS does not document the internal forwarding behaviour, and it is not observable from outside. What is documented — and what you can state in an interview without being wrong — is the requirement: a public NAT Gateway must sit in a subnet whose route table sends internet-bound traffic to an internet gateway.

This is why every check on the ticket passed. Each individual object is healthy. The defect is the relationship between two objects — a NAT Gateway's subnet and that subnet's default route — and no single describe call surfaces it.

Why a timeout and not a refusal#

This is the detail worth internalizing. A dropped packet inside VPC routing produces no ICMP error — the packet is discarded silently. Your client keeps retransmitting SYNs until it gives up.

That signature is diagnostic:

A timeout means "the packet died in flight." Once you see one, you are looking for a routing or filtering problem, not an application problem.

The fix#

Move the NAT Gateway into the public subnet — the one whose route table points 0.0.0.0/0 at the internet gateway. The private route table does not change at all.

Apply it:

terraform apply
# aws_nat_gateway.lab must be replaced
# ...
# Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Verify#

Re-run the exact command that failed. Same shell, same target:

curl -sS --max-time 10 https://checkip.amazonaws.com
# 52.204.18.117    ← the NAT Gateway's Elastic IP

The address returned is the NAT Gateway's EIP, which confirms the traffic is being translated and egressing as intended rather than taking some other path.

Senior debrief#

The transferable lesson: a NAT Gateway is a subnet-resident appliance, not a service endpoint. Anything that forwards traffic on your behalf — NAT Gateway, NAT instance, a firewall appliance, a Transit Gateway attachment — routes according to the route table of the subnet it lives in. When something in the middle of a path is not forwarding, check the middlebox's own routing before you re-read the client's.

Why the team's checklist failed. Every item they verified was a property of a single object, and each one was genuinely fine. The bug lived in a relationship between two objects. This is the most common shape of a cloud networking defect, and it is why "everything looks right" is a signal to widen the aperture rather than to look harder at the same four things.

How this presents in an interview. Asked to debug outbound failures from a private subnet, a weak answer lists things to check. A strong answer establishes the failure mode first — timeout versus refusal versus TLS error — because that single observation eliminates most of the list. Then it walks the path in order and names what forwards at each hop.

Guardrails worth adding. Position is not something you want to rely on review to catch:

• A tflint or Sentinel rule asserting aws_nat_gateway.subnet_id references a subnet tagged Tier = "public".
• In production, one NAT Gateway per availability zone, each in that AZ's public subnet. A single shared NAT Gateway is both a single point of failure and a source of cross-AZ data processing charges.
• Or sidestep the whole class of defect with a regional NAT Gateway. It needs no public subnet to live in and expands across zones on its own, so neither the misplacement in this lab nor the single-zone dependency above is expressible. The exception is private NAT, which regional mode does not support.
• A synthetic egress check from the private tier in CI. This failure is invisible until real traffic needs it, which in this case was during a settlement run.

Cost footnote. Because the NAT Gateway had a public Elastic IP and reported available, it was billing normally for hours while forwarding nothing. Broken infrastructure is rarely free infrastructure.

Clean up#

terraform destroy

Confirm the NAT Gateway and all three interface endpoints are gone — they are the line items that accrue hourly.