Skip to content
VenerableNetworks
Networking4 min read

A stale static route silently shadows the BGP route you are advertising

VPC route priority puts static routes above propagated ones at the same prefix. So a leftover static route wins over a healthy BGP advertisement, and the routing table looks correct because the route is there.

· Venerable Networks

A VPC route table can contain routes from more than one source. Some you wrote. Some arrived by propagation from a virtual private gateway. When two of them cover the same prefix, one wins, and the rule is not the one people assume.

At the same prefix, a static route is preferred over a propagated route. Longest prefix match still applies first — a more specific route beats a less specific one regardless of origin — but at equal specificity, static wins.

Which means a static route left behind from a migration, a test, or a temporary workaround will quietly outrank the BGP advertisement that is supposed to be authoritative.

The failure

On-premises advertises 10.50.0.0/16 over a Site-to-Site VPN. It propagates into the VPC route table. Traffic flows.

Months earlier, during a cutover, someone added a static route for 10.50.0.0/16 pointing at a different target — an old VPN connection, a peering, an appliance ENI that has since been replaced. Nobody removed it.

Everything works until the day the static target stops being correct. Then traffic for that prefix goes to the wrong place, and the route table shows an active route for 10.50.0.0/16, because it does. The propagated route is present too and is simply not the one being used.

The failure is not "no route". It is "the wrong route, and both are there".

Why it survives review

Reading a route table shows you destinations and targets. It does not put "this one is being ignored" next to the loser. You have to notice that two entries cover the same prefix and know the precedence rule to work out which is live.

The Origin field is the thing to read, and it is easy to skip:

aws ec2 describe-route-tables --route-table-ids "$RTB" \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,Origin,GatewayId,State]' \
  --output table

CreateRoute means someone wrote it. EnableVgwRoutePropagation means it arrived by propagation. When both appear for one prefix, the CreateRoute one is winning.

The audit query

Duplicated prefixes across origins are worth finding proactively, because each one is a decision somebody made implicitly:

aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC" \
  --query 'RouteTables[].Routes[].[DestinationCidrBlock,Origin]' --output text \
  | sort | awk '{print $1}' | uniq -d

Anything printed has more than one route. Some of those are legitimate — an IPv4 and IPv6 route share nothing, and a deliberately more specific static route over a broad propagated one is a normal traffic engineering technique. The ones to remove are the same prefix from two origins where nobody remembers writing the static one.

Transit Gateway has its own order

If you terminate on a Transit Gateway instead, the precedence is different and worth knowing separately. In a Transit Gateway route table:

  1. Most specific route wins.
  2. Static routes are preferred over propagated routes.
  3. Among propagated routes, Transit Gateway Connect routes are preferred over VPN routes.

The first two match the VPC behaviour. The third has no VPC equivalent and matters if you run both Connect attachments and VPNs to the same destinations during a migration — the Connect path wins, which is usually what you want and is worth knowing before you rely on the VPN as a fallback.

There is also a related trap on the VPN side: EnableVgwRoutePropagation propagates static routes from the VPN connection. If you configured a static-routing VPN and expected dynamic behaviour, or vice versa, the propagation behaviour differs from what you are picturing.

The operational habit

Two things are worth building in.

Treat static routes as having an owner and an expiry. A static route added during a cutover should be removed in the same change that completes the cutover, and if that is not possible it should be recorded somewhere with the condition for removal. "Temporary" routes are the single most common source of this failure, and the temporary period is measured in months.

Alarm on the count of static routes, not just their contents. A route table whose static route count grows over time is accumulating decisions nobody is tracking. It is a cheap metric and it catches the accumulation before the accumulation causes an incident.

Route priority explains one class of hybrid routing problem. The other big one is asymmetry — where the forward path and the return path are governed by different tables and only one of them is correct. Those two account for most of the "the VPN is up and traffic does not flow" tickets, and they need different diagnostics: priority problems show a route that is present and wrong, asymmetry problems show a request arriving and no reply.

The hybrid connectivity guide covers the precedence rules for both VPC and Transit Gateway route tables alongside the asymmetry case. If you want to watch a propagation problem produce a working request and a lost reply, the VPN route propagation lab is that failure with a virtual private gateway, and the propagation gap lab is the Transit Gateway version.

TaggedAWSBGPRoute Tables