Lab 02: The Hardened Subnet That Broke Everything

What you are working with#

Two subnets in one VPC. No NAT Gateway, no internet gateway on the data side, no Transit Gateway. The failing flow never leaves the VPC.

Scope and constraints#

• In scope: network ACLs, security groups, and the difference between them.
• Out of scope: the database itself, DNS, routing, and the application. The listener works and accepts connections.
• The security finding behind CHG-2291 is valid. Reverting to allow-all is not the answer — find a configuration that is both tight and correct.
• You get a shell on the app tier host only. The data host has no route off the VPC, so Session Manager cannot reach it. That constraint is deliberate and it is part of the lesson.

Deploy the broken state#

terraform init
terraform apply

# Shell on the app tier
terraform output -raw start_session_command

# The target
terraform output -raw data_private_ip

Full source: main.tf. It provisions the VPC, both subnets and their NACLs, both security groups, VPC Flow Logs to CloudWatch, and two instances. The data host runs a small Python TCP listener on 5432 — Python because that subnet has no egress to install anything with.

Allow about two minutes after apply for the listener to start and for flow log records to begin appearing. Flow logs aggregate on a 60-second interval here, so the first records are not instant.

Open a shell on the app tier host:

aws ssm start-session --region us-east-1 --target "$(terraform output -raw app_instance_id)"

Set the target once so the rest of the commands are copy-paste:

DB=10.0.11.20   # use: terraform output -raw data_private_ip
PORT=5432

Confirm the failure#

nc -vz -w 8 "$DB" "$PORT"
# Ncat: Version 7.93 ( https://nmap.org/ncat )
# Ncat: Connection timed out.

A timeout, not a refusal. Nothing rejected the connection — it went unanswered.

Worth contrasting against a port that is genuinely closed, so you can feel the difference:

# Nothing is listening on 9999, and the NACL denies it inbound too.
nc -vz -w 8 "$DB" 9999
# Ncat: Connection timed out.

Both time out, which tells you the NACL is dropping rather than rejecting. A security group or NACL deny silently discards; it never sends a RST.

Watch the conversation from the client#

This is the step that narrows it:

sudo tcpdump -ni any "host $DB and tcp port $PORT" &
nc -vz -w 8 "$DB" "$PORT"
 
# 14:31:02.118 IP 10.0.0.50.41288 > 10.0.11.20.5432: Flags [S], seq 2846...
# 14:31:03.121 IP 10.0.0.50.41288 > 10.0.11.20.5432: Flags [S], seq 2846...
# 14:31:05.129 IP 10.0.0.50.41288 > 10.0.11.20.5432: Flags [S], seq 2846...

SYNs going out, nothing coming back. Note the client's source port: 41288. Hold onto that.

Rule out the security groups#

Confirm the ticket's claim rather than trusting it:

aws ec2 describe-security-groups \
  --filters "Name=tag:Lab,Values=02-nacl-return-path" \
  --query 'SecurityGroups[].{Name:GroupName,In:IpPermissions[].FromPort,Out:IpPermissionsEgress[].IpProtocol}' \
  --output table

The database group permits 5432 from the app group, and both permit all egress. Security groups are stateful, so an allowed outbound connection has its return traffic permitted automatically. No inbound rule is needed on the client for the reply. The ticket was right.

Read the network ACL#

aws ec2 describe-network-acls \
  --network-acl-ids "$(terraform output -raw data_network_acl_id)" \
  --query 'NetworkAcls[0].Entries[].[RuleNumber,Egress,Protocol,PortRange.From,PortRange.To,CidrBlock,RuleAction]' \
  --output table

You will get something like this. Egress: True means an outbound rule:

Read the outbound rules against the source port you noted from tcpdump.

Get the proof from VPC Flow Logs#

Neither host can show you a NACL decision — the drop happens at the subnet boundary, above the instance. Flow logs are the only place it surfaces.

aws logs filter-log-events \
  --log-group-name "$(terraform output -raw flow_log_group)" \
  --start-time "$(( ($(date +%s) - 600) * 1000 ))" \
  --filter-pattern '"5432"' \
  --query 'events[].message' --output text | tail -20

The format is srcaddr srcport dstaddr dstport protocol packets action log-status. Look for two records describing the same connection, and compare their actions.