Lab 15: The Half Of The Internet That Was Missing
IPv6 was enabled on the VPC. Instances have global IPv6 addresses and an IPv6 default route. IPv4 works, IPv6 connections time out, and every dual-stack destination is quietly slower than it was last week.
- Debugging time
- ~20 min
- Reading time
- 12 min
- Reported by
- Integrations
- Tier
- Associate
Partner API over IPv6 times out; unrelated services got slower the same day
Reported by Integrations
We enabled IPv6 on this VPC last week so we could reach a partner API that publishes AAAA records only. Calls to it time out.
IPv6 is definitely on. The VPC has an IPv6 CIDR, the subnet has one, and the instances have real global
IPv6 addresses — I can see them in ip addr. The instance even has an IPv6 default route. Security
groups allow all egress on both families. The internet gateway is attached.
Second thing, possibly unrelated: since the change, a few of our other outbound calls have been slightly slower. Not failing, just slower. Maybe a couple of hundred milliseconds. Nobody can explain it and it might be noise.
What you are working with
One dual-stack VPC, one Availability Zone.
The subnet carries both address families and the internet gateway carries both. The route table has one default route. Count the families, then count the default routes.
| Resource | Configuration |
|---|---|
| VPC | 10.130.0.0/16 and an Amazon-provided IPv6 /56 |
| Subnet | 10.130.1.0/24 and an IPv6 /64, both auto-assigned on launch |
| Internet gateway | Attached |
| Route table | 0.0.0.0/0 → internet gateway |
| Security group | All egress on 0.0.0.0/0 and ::/0 |
| Instances | Global IPv6 addresses, IPv6 default route from the VPC router |
Read the route table row once more and note what is not in it.
- EC2
Client instance
has a global IPv6 address and an IPv6 default route
- FILTER
Security group egress
::/0 permitted
- RTB
Subnet route table
IPv6 routes: the local /56 only
Dropped — No ::/0 route exists, so no route matches an off-VPC IPv6 destination and the packet is discarded. The guest OS has already committed to IPv6 because the VPC router advertised a default route.
- GW
Internet gateway
handles IPv6 fine, never reached
The instance is right that it has a default route. That route is the guest's, learned by router advertisement. The VPC route table is a separate thing and has no IPv6 default.
Scope and constraints
- In scope: why IPv6 traffic does not leave the VPC.
- Out of scope: security groups, the internet gateway, address assignment, and DNS. All correct.
- The instances really do have global IPv6 addresses and an IPv6 default route. The reporter is right.
- The "unrelated" slowness is not unrelated. Explaining it is part of the lab.
Deploy the broken state
cd lab-15-ipv6-default-route
terraform init
terraform apply
aws ssm start-session --target "$(terraform output -raw client_instance_id)"Session Manager works throughout, because the agent talks to AWS over IPv4.
The rest of this lab is part of Labs Pro
The reproduction steps, the deployable Terraform, and the debrief with root cause and remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The incident brief is open, so you can see what you would be solving before you decide.
Already bought it? Sign in and it unlocks.
The rest of this lab is part of Labs Pro
The reproduction steps, the deployable Terraform, and the debrief with root cause and remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The incident brief is open, so you can see what you would be solving before you decide.
Already bought it? Sign in and it unlocks.