Lab 23: The Rule That Was Removed And The Session That Stayed Open
Two identical hosts, two identical long-lived connections, one change that removes the inbound rule admitting the client from both at the same instant. One connection dies. The other keeps working with no rule permitting it, and will for days. The difference is one word in the rule that was removed.
- Debugging time
- ~25 min
- Reading time
- 14 min
- Reported by
- Cloud Security
- Tier
- Professional
Security group change to revoke a client took effect on one host and not the other; the groups are now identical
Reported by Cloud Security
We revoked a client's access to two service hosts this afternoon as part of an offboarding. One Terraform apply, removing the inbound rule for the service port from both hosts' security groups at the same time. Apply succeeded, both rules are gone, I have checked the console on both groups.
The client's connection to the first host dropped immediately, as expected. Its connection to the second host is still open and still working. I can see traffic on it. It has been forty minutes.
Both security groups now have zero inbound rules. Neither permits anything from this client. The client has not reconnected — it is the same TCP session from before the change. I do not understand how a security group with no inbound rules is passing inbound traffic, and I need to know whether this is a bug before I file it as a security incident.
What you are working with
One VPC, one subnet, three instances.
Three hosts in one subnet. Both servers run the same echo service on port 5000 and both admit the client on it — one by naming the client's address, one by admitting the port from anywhere. Both have an all-outbound egress rule. From the client's point of view the two connections are indistinguishable, and from the console both rules do the same job. The labels on the servers give the answer away, which is deliberate: the lab is about why, not which.
| Resource | Configuration |
|---|---|
| Client | 10.230.1.50, outbound-only security group |
| Server TRACKED | 10.230.1.60, inbound TCP 5000 from 10.230.1.50/32, outbound all |
| Server UNTRACKED | 10.230.1.70, inbound TCP 5000 from 0.0.0.0/0, outbound all |
| Service | ncat echo listener on 5000, both servers |
| Instance type | t3.micro — Nitro, not Nitro v6, which matters for the timeout |
| The change | Remove the inbound 5000 rule from both groups in one apply |
Nothing else differs. Same subnet, same image, same service, same client, same change.
- EC2
Client
next packet on the existing TCP connection
- FILTER
Server security group, inbound
zero inbound rules
- RTB
Connection state lookup
is this packet part of a connection the group is tracking?
Dropped — For one server the answer is yes, and the packet is admitted as part of an established connection with no rule consulted. For the other the answer is no, because that flow was never tracked, so the packet is evaluated against the rules — of which there are none.
- DEST
Echo service
answers, or never hears the packet
The decisive hop is not the rule evaluation. It is the lookup before it, which asks whether the security group already knows about this connection. One flow is in that table and one is not, and the rule change is irrelevant to the one that is.
Scope and constraints
- In scope: why a security group with no inbound rules continues to pass one existing connection and not another, when the two connections were admitted by rules that did the same thing.
- Out of scope: the client, the service, the subnet, network ACLs, and the Terraform. All correct.
- The reporter is right that both groups are now identical and that neither admits the client. Both facts are true. One connection still works. Resolve that before you call it a bug.
- Nothing is broken. This is a security finding: a revocation that did not revoke.
- The fix is not a security group change. Pay attention to why.
Deploy the broken state
cd lab-23-security-group-connection-tracking
terraform init
terraform applyUnder three minutes. The state deployed is the before — both rules present, both connections possible. The change that produces the incident is made in the Reproduction, by you, so you are watching when it happens.
TRACKED=$(terraform output -raw tracked_server_ip)
UNTRACKED=$(terraform output -raw untracked_server_ip)
SG_T=$(terraform output -raw tracked_security_group_id)
SG_U=$(terraform output -raw untracked_security_group_id)Confirm the two rules
aws ec2 describe-security-group-rules \
--filters "Name=group-id,Values=$SG_T,$SG_U" \
--query 'SecurityGroupRules[?IsEgress==`false`].[GroupId,FromPort,CidrIpv4]' --output table
# ------------------------------------------------
# | sg-0a… (tracked) | 5000 | 10.230.1.50/32 |
# | sg-0b… (untracked) | 5000 | 0.0.0.0/0 |
# ------------------------------------------------One admits the client by address. One admits the port from anywhere. For a client at 10.230.1.50,
both rules permit the connection.
Open two long-lived connections
Open a session on the client — terraform output -raw start_session_command — and in it, open a
connection to the tracked server:
ncat 10.230.1.60 5000Type a line and press return. The echo server sends it back. Leave this running.
Open a second session on the client and connect to the untracked server:
ncat 10.230.1.70 5000Type a line. It echoes. Leave this running too. You now have two established TCP connections, both working, one to each server.
Make the change
From your workstation, remove the inbound rule from both groups at once:
terraform apply -var='client_rule_present=false' -auto-approveConfirm both rules are gone:
aws ec2 describe-security-group-rules \
--filters "Name=group-id,Values=$SG_T,$SG_U" \
--query 'SecurityGroupRules[?IsEgress==`false`]' --output json
# []Zero inbound rules on either group.
Watch the two sessions
In the session connected to the untracked server (10.230.1.70), type a line and press return.
Nothing comes back. Within a few seconds ncat exits, or the next keystroke produces a connection
reset. The connection is dead. The rule went, the flow went with it.
In the session connected to the tracked server (10.230.1.60), type a line and press return.
It echoes. Type another. It echoes. The connection is alive and passing traffic in both directions through a security group with no inbound rules. Leave it and come back in ten minutes: still alive. It will stay alive for as long as it carries traffic, and for five days after it stops.
Prove it is the same connection
On the tracked server — aws ssm start-session --target <tracked instance> — confirm the socket is the
original one and not a reconnect:
ss -tn state established '( sport = :5000 )'
# Recv-Q Send-Q Local Address:Port Peer Address:Port
# 0 0 10.230.1.60:5000 10.230.1.50:41822Note the client port. It is the one the client opened before the change. No new connection has been made, and a new one would be refused:
# From the client, a THIRD connection to the tracked server:
ncat -w 3 10.230.1.60 5000
# Ncat: Connection timed out.New connections are blocked. The existing one is not. That is the whole finding.
This debrief is part of Labs Pro
The root-cause analysis, packet-flow walkthrough, and Terraform remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The brief, the reproduction steps, and the Terraform stay free — you can still solve this one yourself.
Already bought it? Sign in and it unlocks.