Skip to content
VenerableNetworks
AssociateFree accountNACL / Security GroupsAWS VPC

Lab 02: The Hardened Subnet That Broke Everything

A security review tightened the data tier NACL to permit only the database port. Inbound looks correct, the security groups allow the traffic, both hosts are healthy — and every connection now hangs. Neither host can see why.

Debugging time
~25 min
Reading time
14 min
Reported by
Data Platform
Tier
Associate
INC-1088SEV-2InvestigatingOpened Today 14:02 UTC

Reporting service cannot reach the database after a NACL hardening change

Reported by Data Platform

Environment
staging
Region
us-east-1
Client subnet
10.0.0.0/24 (app-a)
Target subnet
10.0.11.0/24 (data-a)
Change ref
CHG-2291 — NACL tightening

Security review flagged that the data tier subnet had a permissive network ACL. We shipped CHG-2291 last night to restrict it to just the database port.

Since then the reporting service cannot connect. It hangs on connect and times out. We have not changed the application, the security groups, or any routing.

What we verified before escalating:

  • The data tier NACL explicitly allows inbound TCP 5432 from the VPC CIDR. That is the port.
  • The database security group allows 5432 from the app tier security group.
  • The app tier security group allows all outbound.
  • Both instances are running and both subnets are in the same VPC and AZ.
  • The database process is listening — ss -lntp on it showed 0.0.0.0:5432 before we lost access.

We are being asked to roll back CHG-2291, but nobody can explain what is actually wrong with it, and the security finding is legitimate. We would rather understand it than revert blindly.

What you are working with

Two subnets in one VPC. No NAT Gateway, no internet gateway on the data side, no Transit Gateway. The failing flow never leaves the VPC.

Starting state — the flow that fails stays inside the VPC
VPC 10.0.0.0/16 Attached: Internet gateway. Availability Zone A contains app subnet 10.0.0.0/24, default route 0.0.0.0/0, holding App instance, nacl-app; data subnet 10.0.11.0/24, default route local only, holding Data instance, nacl-data.VPC 10.0.0.0/16Availability Zone Aapp subnetpublic10.0.0.0/24App instance10.0.0.50nacl-appallow all both waysdata subnetisolated10.0.11.0/24local onlyData instance10.0.11.20:5432nacl-datatcp/5432 in and out0.0.0.0/0tcp/5432Internet gateway

The connection under investigation is the local one between the two subnets. Only the data subnet has an explicit network ACL, and both of its rules name the service port.

10.0.0.50:41288 → 10.0.11.20:5432 (TCP SYN)
  1. EC2

    Reporting service

    i-… in subnet-app-a (10.0.0.0/24)

  2. FILTER

    App subnet NACL — outbound

    allow all

  3. RTB

    VPC local route

    10.0.0.0/16 → local

  4. FILTER

    Data subnet NACL — inbound

    rule 100: allow tcp 5432 from 10.0.0.0/16

  5. FILTER

    Database security group

    allow 5432 from sg-app

  6. DEST

    Database host

    10.0.11.20:5432 — listening

The outbound path is clean and every check on the ticket confirms it. The request arrives. Trace the reply instead.

Scope and constraints

  • In scope: network ACLs, security groups, and the difference between them.
  • Out of scope: the database itself, DNS, routing, and the application. The listener works and accepts connections.
  • The security finding behind CHG-2291 is valid. Reverting to allow-all is not the answer — find a configuration that is both tight and correct.
  • You get a shell on the app tier host only. The data host has no route off the VPC, so Session Manager cannot reach it. That constraint is deliberate and it is part of the lesson.

Deploy the broken state

main.tf8 lines

Download the file below into an empty directory and apply it. Two t3.micro instances, no NAT Gateway, no interface endpoints — this lab is deliberately cheap to run.

terraform init
terraform apply

# Shell on the app tier
terraform output -raw start_session_command

# The target
terraform output -raw data_private_ip

Full source: main.tf. It provisions the VPC, both subnets and their NACLs, both security groups, VPC Flow Logs to CloudWatch, and two instances. The data host runs a small Python TCP listener on 5432 — Python because that subnet has no egress to install anything with.

Allow about two minutes after apply for the listener to start and for flow log records to begin appearing. Flow logs aggregate on a 60-second interval here, so the first records are not instant.