Lab 07: The Tunnel That Was Up And Went Nowhere
A Site-to-Site VPN reports available, the tunnel reports UP, and the static route for the on-premises prefix is listed on the VPN connection. No traffic passes in either direction, and every page in the VPN console is green.
- Debugging time
- ~35 min
- Reading time
- 12 min
- Reported by
- Network Engineering
- Tier
- Specialty
New Site-to-Site VPN shows both endpoints healthy but carries no traffic
Reported by Network Engineering
The VPN to the branch site was built yesterday. Connection state is available, the tunnel is UP,
and the static route for 192.168.0.0/16 is configured on the connection — we can see it in the
console.
Nothing works. The branch cannot reach the application host in the VPC, and the application host cannot reach anything at the branch. It is not slow or intermittent; it is silent in both directions.
We have checked the customer gateway address, the pre-shared key, and the security groups. The
security group on the AWS side permits the entire branch prefix on all protocols, so it is not that.
The tunnel would not be UP if the IPsec configuration were wrong.
What you are working with
Two VPCs in one account. The second one stands in for the branch office: it runs a Libreswan host as the customer gateway, so the whole path is reproducible without hardware.
Read the two route tables against each other. The branch side sends 10.50.0.0/16 at the IPsec host's interface so the daemon can encapsulate it. The AWS side's table holds a default route for tooling and its implicit local route, and nothing else — the virtual private gateway is attached to the VPC and no route table on this diagram references it.
- EC2
Application host (AWS VPC)
10.50.1.x — replying to 192.168.1.x
- RTB
Subnet route table
local 10.50.0.0/16, default → internet gateway
Dropped — no route for 192.168.0.0/16 at all — the virtual private gateway is attached but nothing propagates its routes into this table
- GW
Virtual private gateway
attached, tunnel UP — never consulted
- DEST
Branch host
192.168.1.x
The tunnel's health is irrelevant to this hop. A route table decides whether traffic is ever handed to the gateway, and this one has no entry for the branch.
| Resource | Configuration |
|---|---|
| AWS VPC | 10.50.0.0/16, subnet 10.50.1.0/24 |
| Branch VPC (simulated) | 192.168.0.0/16, subnet 192.168.1.0/24 |
| Virtual private gateway | Attached to the AWS VPC |
| VPN connection | ipsec.1, static routes only, one tunnel configured |
| VPN connection route | 192.168.0.0/16 — present |
| AWS route table | local plus a default route to the internet gateway |
sg-aws | All protocols from 192.168.0.0/16 |
The branch side is a second VPC with an IPsec daemon on one instance. It is scaffolding, not the subject.
Scope and constraints
- In scope: why traffic does not pass while the connection and tunnel report healthy.
- Out of scope: the pre-shared key, the customer gateway address, IKE parameters, and both security groups. The security group on the AWS side deliberately permits everything from the branch prefix so it cannot be a second cause.
- Static routing, not BGP, so there is no BGP session to blame and no route advertisement to count.
- This lab does not need a working tunnel to diagnose. The evidence is all in the AWS API. If your tunnel is down, you can still complete the diagnosis and the fix; you just cannot watch traffic flow at the end.
Deploy the broken state
cd lab-07-vpn-route-propagation
terraform init
terraform apply
# Shell on the branch host
aws ssm start-session --target "$(terraform output -raw onprem_instance_id)"AWS_HOST=$(terraform output -raw aws_private_ip)
VPN=$(terraform output -raw vpn_connection_id)
RTB=$(terraform output -raw aws_route_table_id)Confirm the failure
From the branch host:
nc -vz -w 8 "$AWS_HOST" 8080
# Ncat: Connection timed out.And from the AWS host, in the other direction:
aws ssm start-session --target "$(terraform output -raw aws_instance_id)"
nc -vz -w 8 192.168.1.20 8080
# Ncat: Connection timed out.Silent both ways. A one-directional failure would point at return routing or a stateless filter; a bidirectional silence means the traffic is not being carried at all.
Confirm the VPN really is healthy
This is the part the ticket already did, and it is worth doing again properly, because the answer is genuinely "yes, it is healthy."
aws ec2 describe-vpn-connections --vpn-connection-ids "$VPN" \
--query 'VpnConnections[0].{State:State,Type:Type,Static:Options.StaticRoutesOnly}'
# {
# "State": "available",
# "Type": "ipsec.1",
# "Static": true
# }
aws ec2 describe-vpn-connections --vpn-connection-ids "$VPN" \
--query 'VpnConnections[0].VgwTelemetry[].{Outside:OutsideIpAddress,Status:Status}' \
--output table
# ----------------------------------------
# | DescribeVpnConnections |
# +------------------+-------------------+
# | Outside | Status |
# +------------------+-------------------+
# | 52.x.x.x | UP |
# | 34.x.x.x | DOWN |
# +------------------+-------------------+One tunnel UP is all this lab configures, and one UP tunnel carries traffic. The DOWN second
tunnel is expected here and is not the fault — see the debrief on why that would still be a finding
in production.
Confirm the route exists on the connection
aws ec2 describe-vpn-connections --vpn-connection-ids "$VPN" \
--query 'VpnConnections[0].Routes' --output table
# --------------------------------------------------
# | DescribeVpnConnections |
# +----------------------+-------------+-----------+
# | DestinationCidrBlock | Source | State |
# +----------------------+-------------+-----------+
# | 192.168.0.0/16 | Static | available |
# +----------------------+-------------+-----------+available. The prefix is declared on the VPN connection and AWS considers it usable. Every check
the ticket describes passes.
Read the route table the traffic actually uses
aws ec2 describe-route-tables --route-table-ids "$RTB" \
--query 'RouteTables[0].Routes[].{Dest:DestinationCidrBlock,Target:GatewayId,State:State,Origin:Origin}' \
--output table
# ------------------------------------------------------------------------
# | DescribeRouteTables |
# +----------------+---------------+----------+--------------------------+
# | Dest | Target | State | Origin |
# +----------------+---------------+----------+--------------------------+
# | 10.50.0.0/16 | local | active | CreateRouteTable |
# | 0.0.0.0/0 | igw-0a1b2c3d | active | CreateRoute |
# +----------------+---------------+----------+--------------------------+There is no 192.168.0.0/16. The route exists on the VPN connection and does not exist in the route
table that the subnet actually consults.
Confirm that propagation is the reason, rather than a route that failed to install:
aws ec2 describe-route-tables --route-table-ids "$RTB" \
--query 'RouteTables[0].PropagatingVgws'
# []Empty. No virtual private gateway is propagating into this table.
This debrief is part of Labs Pro
The root-cause analysis, packet-flow walkthrough, and Terraform remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The brief, the reproduction steps, and the Terraform stay free — you can still solve this one yourself.
Already bought it? Sign in and it unlocks.