Lab 02: The Hardened Subnet That Broke Everything

What you are working with#

Two subnets in one VPC. No NAT Gateway, no internet gateway on the data side, no Transit Gateway. The failing flow never leaves the VPC.

Scope and constraints#

• In scope: network ACLs, security groups, and the difference between them.
• Out of scope: the database itself, DNS, routing, and the application. The listener works and accepts connections.
• The security finding behind CHG-2291 is valid. Reverting to allow-all is not the answer — find a configuration that is both tight and correct.
• You get a shell on the app tier host only. The data host has no route off the VPC, so Session Manager cannot reach it. That constraint is deliberate and it is part of the lesson.

Deploy the broken state#

terraform init
terraform apply

# Shell on the app tier
terraform output -raw start_session_command

# The target
terraform output -raw data_private_ip

Full source: main.tf. It provisions the VPC, both subnets and their NACLs, both security groups, VPC Flow Logs to CloudWatch, and two instances. The data host runs a small Python TCP listener on 5432 — Python because that subnet has no egress to install anything with.

Allow about two minutes after apply for the listener to start and for flow log records to begin appearing. Flow logs aggregate on a 60-second interval here, so the first records are not instant.

Open a shell on the app tier host:

aws ssm start-session --region us-east-1 --target "$(terraform output -raw app_instance_id)"

Set the target once so the rest of the commands are copy-paste:

DB=10.0.11.20   # use: terraform output -raw data_private_ip
PORT=5432

Confirm the failure#

nc -vz -w 8 "$DB" "$PORT"
# Ncat: Version 7.93 ( https://nmap.org/ncat )
# Ncat: Connection timed out.

A timeout, not a refusal. Nothing rejected the connection — it went unanswered.

Worth contrasting against a port that is genuinely closed, so you can feel the difference:

# Nothing is listening on 9999, and the NACL denies it inbound too.
nc -vz -w 8 "$DB" 9999
# Ncat: Connection timed out.

Both time out, which tells you the NACL is dropping rather than rejecting. A security group or NACL deny silently discards; it never sends a RST.

Watch the conversation from the client#

This is the step that narrows it:

sudo tcpdump -ni any "host $DB and tcp port $PORT" &
nc -vz -w 8 "$DB" "$PORT"
 
# 14:31:02.118 IP 10.0.0.50.41288 > 10.0.11.20.5432: Flags [S], seq 2846...
# 14:31:03.121 IP 10.0.0.50.41288 > 10.0.11.20.5432: Flags [S], seq 2846...
# 14:31:05.129 IP 10.0.0.50.41288 > 10.0.11.20.5432: Flags [S], seq 2846...

SYNs going out, nothing coming back. Note the client's source port: 41288. Hold onto that.

Rule out the security groups#

Confirm the ticket's claim rather than trusting it:

aws ec2 describe-security-groups \
  --filters "Name=tag:Lab,Values=02-nacl-return-path" \
  --query 'SecurityGroups[].{Name:GroupName,In:IpPermissions[].FromPort,Out:IpPermissionsEgress[].IpProtocol}' \
  --output table

The database group permits 5432 from the app group, and both permit all egress. Security groups are stateful, so an allowed outbound connection has its return traffic permitted automatically. No inbound rule is needed on the client for the reply. The ticket was right.

Read the network ACL#

aws ec2 describe-network-acls \
  --network-acl-ids "$(terraform output -raw data_network_acl_id)" \
  --query 'NetworkAcls[0].Entries[].[RuleNumber,Egress,Protocol,PortRange.From,PortRange.To,CidrBlock,RuleAction]' \
  --output table

You will get something like this. Egress: True means an outbound rule:

| Rule | Egress | Proto | Ports | CIDR | Action | | --- | --- | --- | --- | --- | --- | | 100 | False | tcp (6) | 5432–5432 | 10.0.0.0/16 | allow | | 32767 | False | all (-1) | all | 0.0.0.0/0 | deny | | 100 | True | tcp (6) | 5432–5432 | 10.0.0.0/16 | allow | | 32767 | True | all (-1) | all | 0.0.0.0/0 | deny |

Read the outbound rules against the source port you noted from tcpdump.

Get the proof from VPC Flow Logs#

Neither host can show you a NACL decision — the drop happens at the subnet boundary, above the instance. Flow logs are the only place it surfaces.

aws logs filter-log-events \
  --log-group-name "$(terraform output -raw flow_log_group)" \
  --start-time "$(( ($(date +%s) - 600) * 1000 ))" \
  --filter-pattern '"5432"' \
  --query 'events[].message' --output text | tail -20

The format is srcaddr srcport dstaddr dstport protocol packets action log-status. Look for two records describing the same connection, and compare their actions.

Root cause#

The data subnet's NACL permits outbound traffic only on port 5432. The database's reply is not sent to port 5432 — it is sent from 5432 to the client's ephemeral port, 41288. No outbound rule matches, so the implicit deny at rule 32767 discards it.

The flow logs show it precisely. Two records, one connection:

10.0.0.50  41288  10.0.11.20  5432   6  3  ACCEPT  OK
10.0.11.20 5432   10.0.0.50   41288  6  3  REJECT  OK

Read those carefully, because together they are the entire diagnosis:

• Line 1 — the SYN arrives at the data subnet and is ACCEPTed. Inbound rule 100 matched.
• Line 2 — the reply leaves the database host, is evaluated on egress, and is REJECTed.

The database answered. The subnet threw the answer away.

That is why no amount of inspection on either host explains it: from the database's perspective it completed its side of the handshake, and from the client's perspective nothing ever came back.

Stateless versus stateful#

This is the distinction the whole lab exists to teach, and it is the single most common source of "but I allowed it" cloud networking bugs.

| | Security group | Network ACL | | --- | --- | --- | | Connection tracking | Stateful | Stateless | | Return traffic | Permitted automatically | Needs its own explicit rule | | Attached to | An ENI | A subnet | | Rule evaluation | All rules, any match allows | Lowest rule number first, first match wins | | Supports deny | No — allow only | Yes |

CHG-2291 scoped a NACL the way you would correctly scope a security group. On a security group, "allow inbound 5432" is complete: the reply is tracked and permitted. On a NACL, it describes only half of the conversation.

Which direction needs the ephemeral range#

Worth being precise, because it is asymmetric and it trips people up.

The data subnet needs outbound ephemeral because it replies to connections. It does not need inbound ephemeral — every inbound packet in this design is destined for 5432, so inbound rule 100 is already correct.

If the database also initiated outbound connections of its own, the mirror would apply: it would need inbound ephemeral to receive those replies. Which direction needs the ephemeral range depends entirely on who opens the connection.

Ephemeral ranges are not universal:

| Source | Range | | --- | --- | | Linux (net.ipv4.ip_local_port_range) | 32768–60999 | | Windows | 49152–65535 | | NAT Gateway | 1024–65535 | | Elastic Load Balancing, Lambda | 1024–65535 |

Because a subnet does not know what will send traffic through it, 1024–65535 is the range AWS documents for NACLs. Narrowing it to one operating system's default is how you get a bug that only appears after someone adds a Windows host or puts a load balancer in front.

The fix#

Add an outbound rule for the ephemeral range. The inbound rule — the actual point of CHG-2291 — stays exactly as tight as the security review wanted.

terraform apply
# aws_network_acl.data will be updated in-place
# Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

NACL changes take effect immediately. There is nothing to restart.

Verify#

Same shell, same command:

nc -vz -w 8 "$DB" "$PORT"
# Ncat: Connected to 10.0.11.20:5432.
 
# Confirm you are talking to the listener, not just completing a handshake
python3 -c "
import socket
s = socket.create_connection(('$DB', $PORT), timeout=5)
print(s.recv(64).decode().strip())
s.close()"
# vn-lab-02 data tier

Then confirm it in the flow logs — the second record should now be ACCEPT:

aws logs filter-log-events \
  --log-group-name "$(terraform output -raw flow_log_group)" \
  --start-time "$(( ($(date +%s) - 120) * 1000 ))" \
  --filter-pattern '"5432"' \
  --query 'events[].message' --output text | tail -4
 
# 10.0.0.50  41902 10.0.11.20 5432  6 5 ACCEPT OK
# 10.0.11.20 5432  10.0.0.50  41902 6 4 ACCEPT OK

Senior debrief#

The transferable lesson: every stateless filter needs rules for both halves of a conversation. A NACL is the only stateless filter in a VPC, which makes it the only place this class of bug can live — and it is why a rule set that reads correctly can still be wrong. When you see a NACL scoped to specific service ports, check the reply direction before anything else.

Why the checklist failed. Every item on the ticket was true. The inbound rule really did permit 5432. The security groups really were correct. The listener really was up. What nobody checked was the reply, because the mental model in play was the stateful one — and under that model, allowing the request implies allowing the response. The bug was in the model, not the configuration.

Why neither host could reveal it. A NACL decision happens at the subnet boundary, above the instance's network stack. tcpdump on the database would have shown the SYN-ACK being sent, and tcpdump on the client would have shown it never arriving — two captures that are individually consistent and jointly baffling. VPC Flow Logs are the only place the REJECT appears. Any time traffic vanishes between two hosts that both look healthy, flow logs are the next tool, not more packet capture.

How this presents in an interview. "A client can't reach a service, security groups are open, routing is correct" is a standard prompt. A weak answer keeps enumerating things to check. A strong answer asks whether a NACL is involved and whether the return path is permitted, then explains the stateless/stateful distinction unprompted. If you can also name where the evidence lives — flow logs, not tcpdump — you are demonstrably someone who has debugged this in production.

The design guidance, which is the part most people get backwards. NACLs are a poor instrument for port-level policy, precisely because they are stateless. Use them as a coarse, blunt control — blocking a CIDR outright, or providing a subnet-wide backstop. Do port and identity scoping in security groups, where statefulness makes the intent expressible and where you can reference other security groups instead of hardcoding CIDRs.

Read that way, CHG-2291 was solving a real finding with the wrong tool. The tight answer is a security group referencing sg-app, which is what the deployed configuration already does. The NACL added no security this design did not already have, and it introduced an outage.

Guardrails worth adding:

• A policy check rejecting any aws_network_acl with a TCP egress rule that does not cover 1024–65535, unless the subnet is explicitly tagged as having no reply traffic.
• A CloudWatch metric filter on flow logs for REJECT, alarmed on a rate change. This failure was loudly visible in telemetry from the first minute and nobody was watching it.
• Treat "tighten the NACL" as a design review trigger rather than a routine change. The question to ask is whether a security group can express the same intent — it usually can, and statefully.

Clean up#

terraform destroy

Both instances and the log group go with it. Nothing here bills hourly once the instances are terminated.