A stale security group rule allows nothing, until the peering it was written for comes back

A security group rule that references another security group is the best kind of rule. It says "the web tier" instead of a list of addresses, and it stays correct when the web tier scales. It has one dependency that a CIDR rule does not: the thing it points at has to still be there, and still be reachable by the mechanism the reference travels over.

When that dependency breaks, AWS has a word for the rule. It is stale, and the documentation is specific about what happens next.

What a stale rule is#

From the VPC peering guide:

A stale security group rule is a rule that references a deleted security group in the same VPC or in a peer VPC, or that references a security group in a peer VPC for which the VPC peering connection has been deleted. When a security group rule becomes stale, it's not automatically removed from your security group—you must manually remove it.

Peering was the original way to reference a group in another VPC. There are now two more, and the describe-stale-security-groups reference enumerates every way a rule gets stale:

Rules are stale when they reference a deleted security group. Rules can also be stale if they reference a security group in a peer VPC for which the VPC peering connection has been deleted, across a transit gateway where the transit gateway has been deleted (or the transit gateway security group referencing feature has been disabled), or if a security group VPC association has been disassociated.

Four conditions:

The third is the quiet one. Security group referencing across a transit gateway has to be enabled on the gateway and on each attachment, and the attachment documentation notes that disabling it on the gateway disables it on every attachment. One flag flipped on the hub stales every cross-VPC reference in every spoke at once.

What a stale rule does#

Nothing. That is the point and the problem.

A rule that references a security group admits traffic from the private addresses of the network interfaces associated with that group, per the security group rules documentation:

For inbound rules, the EC2 instances associated with a security group can receive inbound traffic from the private IP addresses from the network interfaces for the EC2 instances associated with the referenced security group.

If the referenced group is gone, there are no such interfaces. If the peering or transit gateway path is gone, there is no mechanism to resolve them. Either way the set of permitted sources is empty, and the rule permits no one.

So a stale rule is not a security hole. It is three other things.

It is still a rule. The same page: "A rule that references another security group counts as one rule, no matter the size of the referenced security group." A stale reference holds one slot of the per-group rule quota while matching nothing. A group that has accumulated a dozen of them over years of peering changes is a dozen rules closer to the limit for no reason, and the prefix-list weight lab shows what the limit looks like when you hit it.

It is misleading to read. An audit of the group shows sg-bbbb2222 as an allowed source on port 22. Nothing in describe-security-groups says that source resolves to nothing. The reviewer either trusts the rule and believes access exists that does not, or goes looking for sg-bbbb2222 and finds it deleted, and spends the time working out which.

It is waiting. This is the one that matters.

The rule that comes back#

The peering guide, one sentence after the definition:

If a security group rule is stale because the VPC peering connection was deleted, the rule will no longer be marked as stale if you create a new VPC peering connection with the same VPCs.

Read that as a timeline. In year one, VPC A and VPC B are peered and a rule in A admits SSH from sg-bbbb2222 in B. The project ends, the peering is deleted, the rule goes stale, nobody removes it because stale rules are not removed. In year three, a different team peers A and B for a different purpose. The rule is no longer stale. It is a working rule again, admitting SSH from whatever interfaces are associated with sg-bbbb2222 today, which may be a workload that did not exist when the rule was written, owned by people who have never seen the rule.

Nothing was changed in A's security group. Nobody approved SSH from B. The rule was there the whole time, allowing nothing, and the peering made it allow something.

The same logic applies to the transit gateway case. A rule staled by disabling security group referencing on the gateway is a rule that re-arms when someone re-enables it. The documentation does not state the transit gateway revival explicitly as it does for peering, but the mechanism is the same: the rule was never removed, and staleness is a property of whether the reference can currently be resolved.

Finding them#

The command is per VPC and paginated:

aws ec2 describe-stale-security-groups --vpc-id vpc-aaaaaaaa

Two different things come back looking stale, and the output tells them apart. From the peering guide's example, a rule staled by a deleted peering:

"UserIdGroupPairs": [
    {
        "VpcId": "vpc-bbbbbbbb",
        "PeeringStatus": "deleted",
        "UserId": "123456789101",
        "GroupName": "Prod1",
        "VpcPeeringConnectionId": "pcx-b04deed9",
        "GroupId": "sg-bbbb2222"
    }
]

PeeringStatus: deleted. The group still exists, with a name and an owner; the path to it does not. This is the one that comes back if the VPCs are re-peered.

And from the CLI reference's example, a rule staled by a deleted group:

"UserIdGroupPairs": [
    {
        "VpcId": "vpc-7a20e51f",
        "GroupId": "sg-279ab042",
        "VpcPeeringConnectionId": "pcx-b04deed9",
        "PeeringStatus": "active"
    }
]

PeeringStatus: active, and no UserId or GroupName. The reference documents why: "If the referenced security group is deleted, this value is not returned." The peering is fine; the group on the other end is gone. This one does not come back unless someone recreates a group with the same ID, which they cannot.

Both are stale. The first is the dormant one.

Then remove them with the ordinary revoke commands, which the guide points at directly. A loop over every VPC in a Region:

for vpc in $(aws ec2 describe-vpcs --query 'Vpcs[].VpcId' --output text); do
  aws ec2 describe-stale-security-groups --vpc-id "$vpc" \
    --query 'StaleSecurityGroupSet[].{group: GroupId,
              inbound: StaleIpPermissions[].UserIdGroupPairs[].[GroupId,PeeringStatus],
              outbound: StaleIpPermissionsEgress[].UserIdGroupPairs[].[GroupId,PeeringStatus]}' \
    --output json
done
# [{"group": "sg-5fa68d3a", "inbound": [["sg-bbbb2222", "deleted"]], "outbound": []}]

Anything that prints is a rule to revoke, and anything with "deleted" next to it is a rule to revoke before the next peering request.

The command to run before you delete anything#

Staleness is created on the other side of the reference. You delete a group, or a peering, and rules in groups you may not own go stale. The peering guide gives the reverse lookup:

To determine if your security group is being referenced in the rules of a security group in a peer VPC, use one of the following commands … describe-security-group-references

aws ec2 describe-security-group-references --group-id sg-bbbb2222
{
  "SecurityGroupsReferenceSet": [
    {
      "ReferencingVpcId": "vpc-aaaaaaaa",
      "GroupId": "sg-bbbb2222",
      "VpcPeeringConnectionId": "pcx-b04deed9"
    }
  ]
}

Run that before deleting a security group or tearing down a peering connection, and you have the list of groups that are about to acquire a stale rule. Tell their owners, or remove the rules first. It is the only way to find out, because the referencing side gets no notification. The rules documentation adds a detail that makes this sharper for cross-account references:

When a referenced security group is owned by another account, the owner account does not receive CloudTrail events for these actions.

The account whose group is being referenced does not even see the rule being created. It will not see it go stale either.

Not stale, and still empty#

One more case belongs here because it looks like a working reference and behaves like a stale one. From the rules documentation's limitations:

If you configure routes to forward the traffic between two instances in different subnets through a middlebox appliance, you must ensure that the security groups for both instances allow traffic to flow between the instances. The security group for each instance must reference the private IP address of the other instance or the CIDR range of the subnet that contains the other instance as the source. If you reference the security group of the other instance as the source, this does not allow traffic to flow between the instances.

A rule referencing the other instance's group, on a path that goes through an inspection appliance, is not stale. Both groups exist, nothing was deleted, describe-stale-security-groups returns nothing. It still admits no traffic. The same limitation is listed for transit gateway referencing through a Gateway Load Balancer or Network Firewall: "security group referencing through the transit gateway does not work across AWS Gateway Load Balancer or an AWS Network Firewall." Anyone moving an inspection VPC into the path and keeping their reference-based rules will find the rules intact and the traffic gone.

What to do with it#

Run describe-stale-security-groups on every VPC, on a schedule. It is the only API that reports the condition, it is per-VPC, and the console equivalent is a menu item nobody opens. Treat a non-empty result as a finding.

Revoke PeeringStatus: deleted rules first. They are the ones that re-arm. A rule staled by a deleted group is clutter; a rule staled by a deleted peering is a grant waiting for a path.

Run describe-security-group-references before deleting a group or a peering. The referencing side is not told. You are the only one who can tell them.

Treat "disable referencing on the transit gateway" as a change to every spoke. One flag on the hub stales every cross-VPC reference below it, and re-enabling it brings every one of them back.

Make peering requests check the destination VPC for stale rules before approval. The question "what rules in this VPC reference groups in the VPC we are about to peer with" has a precise answer, and asking it turns the revival from a surprise into a decision.

A stale rule is an allow with nobody on the list. The list is not empty forever. The mechanism that empties it is deleting a path, and the mechanism that fills it again is creating one, and the second thing is done by people who have never read the first.