Lab 25: The IPAM Guardrail That Recorded The Violation
An IPAM pool with a /24 allocation rule, adopted so nobody could carve out a /16 again. Someone carved out a /16. Someone else created a VPC on top of an existing allocation. Both VPCs are up, both were created without a single error, and IPAM has an accurate record of each. The rule was never a gate.
- Debugging time
- ~30 min
- Reading time
- 10 min
- Reported by
- Network Governance
- Tier
- Professional
IPAM allocation rules did not prevent an oversized VPC or an overlapping one; both created without error
Reported by Network Governance
We rolled out IPAM in Q2 specifically to stop address space being consumed by hand. The private pool has a minimum netmask rule of /24 and auto-import is on. The rollout review said this would "prevent noncompliant allocations".
This morning's resource view shows a /16 in the pool marked noncompliant, and a /23 sitting
directly on top of one of our IPAM-allocated /24s. Neither team got an error. Neither used IPAM.
Both VPCs are live with workloads in them.
I need to understand what the rule was doing if it was not doing this, and what actually stops it.
What you are working with
One IPAM, one pool, three VPCs, and a role that stands in for whoever builds VPCs in this account.
The hand-typed /16 is 256 times the size of the allocation the rule permits. The third VPC, a /23 at 10.32.0.0, is not drawn because it sits on top of the /24 and this figure refuses overlaps; its absence here is the finding.
| Resource | Configuration |
|---|---|
| IPAM | Advanced Tier, one operating Region |
| Pool | 10.32.0.0/12, locale = Region, allocation_min_netmask_length = 24, auto_import = true |
VPC compliant | ipv4_ipam_pool_id + netmask 24 → IPAM chose 10.32.0.0/24 |
VPC oversized | cidr_block = "10.41.0.0/16", typed |
VPC overlapping | cidr_block = "10.32.0.0/23", typed, on top of compliant |
Role network-builder | ec2:*, no guardrail |
Scope and constraints
- In scope: what an allocation rule governs, what it does not, and the one mechanism that does refuse a hand-typed CIDR.
- Out of scope: pool hierarchy design, sharing with RAM, cost forecasting. The IPAM guide covers those.
- The reporter's rollout review was wrong, and it was wrong in a way the AWS documentation anticipates. Find the sentence.
- Nothing in this lab is broken in the sense of traffic failing. Both hand-typed VPCs work. This is a governance incident, and the fix is a policy, not a route.
Deploy the broken state
cd lab-25-ipam-rules-are-a-report
terraform init
terraform applyUnder two minutes. IPAM then needs time to discover the hand-typed VPCs and attempt the import. Allow several minutes before the resource view is complete; if a VPC is missing from it, wait and query again.
SCOPE=$(terraform output -raw scope_id)
POOL=$(terraform output -raw pool_id)
ROLE=$(terraform output -raw builder_role_arn)Confirm the rule is in place
Start by proving the reporter right about the configuration, so you stop re-checking it.
aws ec2 describe-ipam-pools --ipam-pool-ids "$POOL" \
--query 'IpamPools[0].[AllocationMinNetmaskLength,AutoImport,Locale]' --output text
# 24 True us-east-1Minimum netmask 24. Auto-import on. Exactly as reviewed.
Read the resource view
This is where the three VPCs tell three different stories.
aws ec2 get-ipam-resource-cidrs --ipam-scope-id "$SCOPE" \
--filters Name=resource-type,Values=vpc \
--query 'IpamResourceCidrs[].[ResourceName,ResourceCidr,ManagementState,ComplianceStatus,OverlapStatus]' \
--output table
# ------------------------------------------------------------------------------
# | vn-lab-25-vpc-compliant | 10.32.0.0/24 | managed | compliant | overlapping |
# | vn-lab-25-vpc-oversized | 10.41.0.0/16 | managed | noncompliant | nonoverlapping |
# | vn-lab-25-vpc-overlapping | 10.32.0.0/23 | unmanaged | unmanaged | overlapping |
# ------------------------------------------------------------------------------Three rows, three combinations. Take them one at a time.
compliant: managed and compliant, because IPAM chose its CIDR and the choice satisfied the rule.
But overlapping, because something was later put on top of it. IPAM is telling you the victim as
well as the offender.
oversized: managed. IPAM imported a /16 that nobody asked it for, because auto-import is
on and the CIDR is inside the pool's range. Then it evaluated the import against the rule, and the
import failed the rule. noncompliant is a verdict on a resource that already exists.
overlapping: unmanaged, even with auto-import on. The documentation for the auto-import option
explains why: "The CIDRs that will be allocated for these resources must not already be allocated to
other resources in order for the import to succeed." The space was already allocated to compliant,
so the import could not happen, so the VPC is not measured against the rule at all. IPAM still
evaluates overlap for unmanaged resources, and reports overlapping. It is the only one of the three
statuses this VPC can earn.
Three VPCs created, zero refused, three accurate reports.
Create a fourth one, as the builder would
The role has ec2:* and nothing else. Assume it and do what the two teams did:
CREDS=$(aws sts assume-role --role-arn "$ROLE" --role-session-name lab25 \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text)
export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | cut -f1)
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | cut -f2)
export AWS_SESSION_TOKEN=$(echo "$CREDS" | cut -f3)
aws ec2 create-vpc --cidr-block 10.42.0.0/16 \
--query 'Vpc.[VpcId,CidrBlock]' --output text
# vpc-0d1e… 10.42.0.0/16Another /16, inside the pool, straight past the rule. Delete it before moving on:
aws ec2 delete-vpc --vpc-id vpc-0d1e…
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKENThis debrief is part of Labs Pro
The root-cause analysis, packet-flow walkthrough, and Terraform remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The brief, the reproduction steps, and the Terraform stay free — you can still solve this one yourself.
Already bought it? Sign in and it unlocks.