AssociateFreeAWS VPCNAT Gateway

Lab 01: The Misplaced NAT Gateway

A batch job in a private subnet cannot reach a third-party API. The route table looks correct, the security groups are wide open, and the NAT Gateway reports Available. Find out why every outbound connection still hangs.

Debugging time: ~20 min
Reading time: 10 min
Reported by: Payments Platform
Tier: Associate

INC-1043SEV-3InvestigatingOpened Today 09:14 UTC

Outbound API calls from the private subnet time out after deploying the new VPC

Reported by Payments Platform

Environment: staging
Region: us-east-1
VPC: 10.0.0.0/16
Affected subnet: 10.0.11.0/24 (private-a)

We cut over the settlement batch job to the new staging VPC last night. The job starts, then every call to our payment processor's API hangs and eventually times out. No response, no TLS error, nothing in the logs except the client-side timeout.

Things we already checked:

The NAT Gateway shows Available in the console with an Elastic IP attached.
The private subnet's route table has 0.0.0.0/0 pointed at the NAT Gateway.
The instance security group allows all outbound traffic.
DNS resolves fine from the instance.

Networking says the config "looks right." We are blocked on the settlement run.

What you are working with

A single-AZ VPC with the usual two-tier layout. One public subnet, one private subnet, an internet gateway, and a NAT Gateway that is supposed to give the private tier outbound access.

10.0.11.x:49820 → 52.94.236.248:443 (TCP SYN, outbound HTTPS)

EC2
Batch instance
i-… in subnet-private-a (10.0.11.0/24)
FILTER
Security group egress
sg-app — allow all outbound (0.0.0.0/0)
FILTER
Network ACL
default NACL — allow all inbound and outbound
RTB
Private route table
rtb-private — 0.0.0.0/0 → nat-…
GW
NAT Gateway
nat-… — state: available, EIP attached
DEST
api.processor.example
52.94.236.248:443

Every hop the team checked passes. The packet reaches the NAT Gateway. What happens after that is the lab.

Scope and constraints

In scope: VPC routing, subnet placement, NAT Gateway configuration.
Out of scope: the third-party API itself, TLS, application code, DNS. The endpoint is reachable from anywhere with working egress.
The instance has no public IP and must stay that way. Assigning one is not the fix.
You have Session Manager access to the instance. It works because the lab provisions interface endpoints for SSM inside the VPC, so your shell does not depend on the broken egress path.

Deploy the broken state

main.tf5 lines

Download

Download the file below into an empty directory, then apply it. The apply succeeds — nothing here is invalid, which is exactly why this class of bug survives code review.

terraform init
terraform apply

# Grab the session command from the outputs
terraform output -raw start_session_command

Full source: main.tf — one file, no modules, no remote state. It provisions a VPC, both subnets, an internet gateway, a NAT Gateway with an Elastic IP, both route tables, SSM interface endpoints, and a t3.micro running Amazon Linux 2023.

Apply takes about three minutes, most of it waiting on the NAT Gateway and the endpoints.

Open a shell on the private instance. Session Manager reaches it through the VPC endpoints, so this works even though outbound internet is broken:

aws ssm start-session --region us-east-1 --target "$(terraform output -raw instance_id)"

Confirm the failure

Once you have a prompt, try to reach anything on the public internet. Use --max-time so you are not waiting on the default two-minute timeout:

curl -sS --max-time 10 https://checkip.amazonaws.com
# curl: (28) Connection timed out after 10001 milliseconds

The connection hangs and then times out. Note the failure mode: a timeout, not a refusal. That distinction matters and it narrows the problem considerably.

# Refused would mean something actively rejected the packet.
# Timed out means nobody answered at all.
curl -sS --max-time 10 -o /dev/null -w '%{http_code}\n' http://example.com
# curl: (28) Connection timed out after 10002 milliseconds

Rule out DNS

DNS is often blamed first. Rule it out explicitly — resolution succeeds, so name resolution is not the blocker:

dig +short checkip.amazonaws.com
# 54.235.98.220
# 3.229.87.221
 
# Connect straight to an IP, no DNS involved. Same timeout.
curl -sS --max-time 10 https://54.235.98.220 -k
# curl: (28) Connection timed out after 10003 milliseconds

Watch the packets leave

Confirm the instance is genuinely transmitting and getting nothing back. Retransmissions with no reply mean the packet is leaving the host and dying somewhere upstream:

sudo tcpdump -ni any 'tcp port 443 and host 54.235.98.220' &
curl -sS --max-time 8 https://54.235.98.220 -k
 
# 09:41:02.118 IP 10.0.11.34.49820 > 54.235.98.220.443: Flags [S], seq 1583...
# 09:41:03.121 IP 10.0.11.34.49820 > 54.235.98.220.443: Flags [S], seq 1583...
# 09:41:05.129 IP 10.0.11.34.49820 > 54.235.98.220.443: Flags [S], seq 1583...

Three SYNs, zero SYN-ACKs. The instance is doing its job. Something between the instance and the internet is swallowing the traffic.

Inspect the path from the outside

Back on your workstation, walk the routing configuration. Start where the team started — the private route table:

aws ec2 describe-route-tables \
  --route-table-ids "$(terraform output -raw private_route_table_id)" \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,NatGatewayId,GatewayId,State]' \
  --output table

The default route is present and active, pointing at the NAT Gateway. The ticket was accurate. Now ask the question nobody on the ticket asked:

# Which subnet is the NAT Gateway actually in?
aws ec2 describe-nat-gateways \
  --nat-gateway-ids "$(terraform output -raw nat_gateway_id)" \
  --query 'NatGateways[0].[State,SubnetId]' --output text
 
# And which subnets are which?
echo "public:  $(terraform output -raw public_subnet_id)"
echo "private: $(terraform output -raw private_subnet_id)"

Compare those three values carefully. That comparison is the whole lab.

Root cause

The NAT Gateway was created in the private subnet — the same subnet as the instances it is meant to serve.

aws ec2 describe-nat-gateways --nat-gateway-ids nat-0f8e… \
  --query 'NatGateways[0].SubnetId' --output text
# subnet-0b91…   ← this is subnet-private-a, not public-a

A NAT Gateway is not a router that magically reaches the internet. It is a managed appliance that lives in a subnet and forwards traffic according to that subnet's route table. Placing one in a private subnet produces a device with a public Elastic IP that has no path to the internet gateway.

The result is a routing loop. Traffic from the instance hits the private route table, gets forwarded to the NAT Gateway, which is itself in the private subnet, so the NAT Gateway consults the same private route table, which sends 0.0.0.0/0 back to itself.

rtb-private (vn-lab-01-rtb-private)assoc: subnet-private-a — and the NAT Gateway lives here too

Destination	Target	State	Origin
10.0.0.0/16	local	active	CreateRouteTable
0.0.0.0/0	nat-0f8e…	activeResolves to a NAT Gateway inside this same subnet, so this route points back at itself.	CreateRoute

AWS reports this route as active because the NAT Gateway exists and is available. Route state describes the target's existence, not whether the path leads anywhere.

This is why every check on the ticket passed. Each individual object is healthy. The defect is the relationship between two objects — a NAT Gateway's subnet and that subnet's default route — and no single describe call surfaces it.

10.0.11.34:49820 → 54.235.98.220:443 (TCP SYN)

EC2
Batch instance
i-… in subnet-private-a
FILTER
Security group egress
sg-app — allow all outbound
RTB
Private route table
0.0.0.0/0 → nat-0f8e…
GW
NAT Gateway (in subnet-private-a)
Performs NAT, then consults its own subnet route table
RTB
Private route table — again
0.0.0.0/0 → nat-0f8e… (itself)
Dropped — no route to an internet gateway from this subnet; the packet has nowhere left to go
DEST
checkip.amazonaws.com
54.235.98.220:443

The packet is translated and then handed back to the route table that sent it. Nothing generates an ICMP unreachable, which is why the client sees a silent timeout instead of an error.

Why a timeout and not a refusal

This is the detail worth internalizing. A dropped packet inside VPC routing produces no ICMP error — the packet is discarded silently. Your client keeps retransmitting SYNs until it gives up.

That signature is diagnostic:

| Symptom | Typical cause | | --- | --- | | Connection refused | Reached the host; nothing listening on that port | | Connection timed out | Packet dropped in transit — routing, NACL, or security group | | TLS or certificate error | Reached the service; the problem is above layer 4 | | DNS resolution failure | Never got as far as routing |

A timeout means "the packet died in flight." Once you see one, you are looking for a routing or filtering problem, not an application problem.

The fix

Move the NAT Gateway into the public subnet — the one whose route table points 0.0.0.0/0 at the internet gateway. The private route table does not change at all.

main.tf+5-1

Diff of main.tf: 5 lines added, 1 lines removed.
1	1		resource "aws_nat_gateway" "lab" {
2	2		allocation_id = aws_eip.nat.id
3		-	subnet_id = aws_subnet.private.id
4	3
	4	+	# A NAT Gateway forwards via its OWN subnet's route table. It must sit in a
	5	+	# subnet with a default route to the internet gateway, never in the private
	6	+	# subnet it serves.
	7	+	subnet_id = aws_subnet.public.id
	8	+
5	9		tags = { Name = "${var.name}-nat" }
6	10
7	11		depends_on = [aws_internet_gateway.lab]
⋯ 1 unchanged line

One attribute. Note that Terraform replaces the NAT Gateway rather than moving it — subnet_id forces replacement, and the new gateway keeps the same Elastic IP.

Apply it:

terraform apply
# aws_nat_gateway.lab must be replaced
# ...
# Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Verify

Re-run the exact command that failed. Same shell, same target:

curl -sS --max-time 10 https://checkip.amazonaws.com
# 52.204.18.117    ← the NAT Gateway's Elastic IP

The address returned is the NAT Gateway's EIP, which confirms the traffic is being translated and egressing as intended rather than taking some other path.

Senior debrief

The transferable lesson: a NAT Gateway is a subnet-resident appliance, not a service endpoint. Anything that forwards traffic on your behalf — NAT Gateway, NAT instance, a firewall appliance, a Transit Gateway attachment — routes according to the route table of the subnet it lives in. When something in the middle of a path is not forwarding, check the middlebox's own routing before you re-read the client's.

Why the team's checklist failed. Every item they verified was a property of a single object, and each one was genuinely fine. The bug lived in a relationship between two objects. This is the most common shape of a cloud networking defect, and it is why "everything looks right" is a signal to widen the aperture rather than to look harder at the same four things.

How this presents in an interview. Asked to debug outbound failures from a private subnet, a weak answer lists things to check. A strong answer establishes the failure mode first — timeout versus refusal versus TLS error — because that single observation eliminates most of the list. Then it walks the path in order and names what forwards at each hop.

Guardrails worth adding. Position is not something you want to rely on review to catch:

A tflint or Sentinel rule asserting aws_nat_gateway.subnet_id references a subnet tagged Tier = "public".
In production, one NAT Gateway per availability zone, each in that AZ's public subnet. A single shared NAT Gateway is both a single point of failure and a source of cross-AZ data processing charges.
A synthetic egress check from the private tier in CI. This failure is invisible until real traffic needs it, which in this case was during a settlement run.

Cost footnote. Because the NAT Gateway had a public Elastic IP and reported available, it was billing normally for hours while forwarding nothing. Broken infrastructure is rarely free infrastructure.

Clean up

terraform destroy

Confirm the NAT Gateway and all four interface endpoints are gone — they are the line items that accrue hourly.