Lab 15: The Half Of The Internet That Was Missing

What you are working with#

One dual-stack VPC, one Availability Zone.

Read the route table row once more and note what is not in it.

Scope and constraints#

• In scope: why IPv6 traffic does not leave the VPC.
• Out of scope: security groups, the internet gateway, address assignment, and DNS. All correct.
• The instances really do have global IPv6 addresses and an IPv6 default route. The reporter is right.
• The "unrelated" slowness is not unrelated. Explaining it is part of the lab.

Deploy the broken state#

cd lab-15-ipv6-default-route
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw client_instance_id)"

Session Manager works throughout, because the agent talks to AWS over IPv4.

Confirm IPv6 looks configured#

Everything the reporter said is true, and this is worth establishing rather than assuming:

ip -6 addr show scope global
# 2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001
#     inet6 2600:1f16:16c6:c501:c8e3:6ae1:44a0:7341/128 scope global dynamic noprefixroute

ip -6 route show default
# default via fe80::ff:daff:fefa:f123 dev ens5 proto ra metric 512 expires 1797sec pref medium

A global address and a default route. The instance is fully prepared to use IPv6 — which is exactly why it does, and why the packets go nowhere.

Note proto ra on that route. The kernel is telling you where the route came from: a router advertisement, sent by the VPC router. Nothing in the VPC route table produced it, and its presence says nothing about whether the VPC can actually forward the traffic. It is worth getting used to reading that field, because a route the guest learned and a route the VPC will honour are different claims.

Confirm IPv6 fails off the VPC#

curl -6 -s -m 8 -o /dev/null -w '%{http_code}\n' https://ipv6.google.com/; echo "exit=$?"
# 000
# exit=28

A timeout, not an error, and the exit code is the precise form of the evidence. 28 is CURLE_OPERATION_TIMEDOUT. Had the kernel lacked a route it would have refused to send and you would get 7 (CURLE_COULDNT_CONNECT) with Network is unreachable. 28 means the packets left the instance and nothing came back — the failure is beyond the guest.

ping6 -c 3 -W 3 2606:4700:4700::1111
# 3 packets transmitted, 0 received, 100% packet loss

Confirm IPv6 works inside the VPC#

This is the step that localises the fault.

PEER=$(terraform output -raw peer_ipv6)
 
ping6 -c 3 "$PEER"
# 3 packets transmitted, 3 received, 0% packet loss
# rtt min/avg/max/mdev = 0.209/0.238/0.294/0.039 ms
 
ncat -6 --recv-only -w 5 "$PEER" 5000
# lab-15-ipv6-in-vpc-ok

IPv6 is working. The address family is configured correctly, the interface is up, the security group permits it, and two hosts can talk over it. What fails is leaving the VPC.

That combination — a family that works locally and not remotely — points at exactly one place.

Explain the "unrelated" slowness#

Time a request to a dual-stack host, first letting the client choose, then forcing each family:

curl -s -o /dev/null -w 'default: %{time_total}s\n' https://www.cloudflare.com/
# default: 0.267s
 
curl -4 -s -o /dev/null -w 'ipv4:    %{time_total}s\n' https://www.cloudflare.com/
# ipv4:    0.067s
 
curl -6 -s -m 8 -o /dev/null -w 'ipv6:    %{time_total}s\n' https://www.cloudflare.com/
# ipv6:    (times out)

The default is 200 ms slower than forcing IPv4. That is Happy Eyeballs doing its job: curl resolves both families, starts the IPv6 attempt first because AAAA is preferred, waits out its happy-eyeballs timeout — 200 ms in libcurl by default — then starts IPv4 in parallel and uses whichever connects.

So every dual-stack destination pays a fixed penalty while the doomed IPv6 attempt is abandoned. The reporter's "couple of hundred milliseconds" is precisely the fingerprint of a broken IPv6 path behind a working IPv4 one.

Prove the delay is the happy-eyeballs timer#

A 200 ms gap is suggestive, not conclusive — plenty of things cost 200 ms. Make the timer itself the variable, which turns the correlation into a demonstration:

curl --happy-eyeballs-timeout-ms 1    -s -o /dev/null -w 'he=1ms:    %{time_total}s\n' https://www.cloudflare.com/
# he=1ms:    0.067s
 
curl --happy-eyeballs-timeout-ms 1000 -s -o /dev/null -w 'he=1000ms: %{time_total}s\n' https://www.cloudflare.com/
# he=1000ms: 1.065s

Setting the head-start to 1 ms removes the penalty entirely and matches forced IPv4. Setting it to 1000 ms grows the penalty to almost exactly one second. The delay tracks the knob, so the delay is the knob — you are watching IPv6 be attempted and abandoned, on a timer you control.

This is worth keeping in your pocket. It is a two-command test that distinguishes "our IPv6 path is broken and Happy Eyeballs is covering for it" from every other cause of a few hundred milliseconds, without touching the network.

This also explains why nothing looked broken. Most destinations are dual-stack, Happy Eyeballs covers them, and only the IPv6-only partner API fails outright.

Read the route table's IPv6 entries#

aws ec2 describe-route-tables --region us-east-1 \
  --route-table-ids "$(terraform output -raw route_table_id)" \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,DestinationIpv6CidrBlock,GatewayId,State]' \
  --output table
# ---------------------------------------------------------------------------
# |  10.130.0.0/16  |  None                     |  local      |  active     |
# |  0.0.0.0/0      |  None                     |  igw-0ef0…  |  active     |
# |  None           |  2600:1f16:16c6:c500::/56 |  local      |  active     |
# ---------------------------------------------------------------------------

Three routes. Two local, one IPv4 default. There is no ::/0.

The IPv6 local route was created automatically when the CIDR was associated, which is why the table looks like it has IPv6 in it. Local routes cover the VPC's own prefixes and nothing else — which is consistent with IPv6 working between the two instances and failing everywhere else.