Professional9 min read

Hybrid Connectivity: VPN, Direct Connect, and the BGP Attributes That Actually Work

Which AWS-side endpoint you terminate on decides your route quotas, whether you can use ECMP, and how failover behaves. Most hybrid incidents trace back to that one choice, a route ceiling that resets BGP, or a return path nobody drew.

Published Mar 4, 2026

Hybrid connectivity looks like two decisions — VPN or Direct Connect, and which prefixes to advertise. In practice the consequential choice is a third one nobody frames as a decision: whether you terminate on a virtual private gateway or a Transit Gateway. That single pick sets your route quotas, determines whether ECMP is available at all, and changes how failover behaves. Get it wrong and you discover the constraint during a migration, at which point moving is a project.

The AWS-side endpoint decides your ceilings

Both a VGW and a Transit Gateway will terminate a Site-to-Site VPN. They are not interchangeable.

| | Virtual private gateway | Transit Gateway | | --- | --- | --- | | Dynamic routes advertised to AWS | 100 | 1,000 | | Routes advertised from AWS | 1,000 | 5,000 | | Static routes to AWS | 100 | — | | ECMP across tunnels | Not supported | Supported, with dynamic routing enabled | | VPCs served by one VPN | One | Every attached VPC |

Quotas per the Site-to-Site VPN quotas page; none of them are adjustable.

The 100-route figure is the one that causes outages, because of how it fails.

This is a bad failure mode for a specific reason: the trigger is usually somewhere else entirely. Nobody changes the VPN. Someone adds a subnet on the corporate network, a redistribution rule picks it up, the advertised count crosses 100, and a link that has been stable for two years starts flapping. The change and the symptom are in different teams' systems.

Mitigations, in descending order of preference: summarize on the customer gateway so you advertise a handful of aggregates instead of every subnet; advertise a default route to AWS if your topology allows it; or terminate on a Transit Gateway, where the ceiling is 1,000.

Propagation is opt-in on a VGW too

If you terminate on a VGW, the VPC route tables do not learn your on-premises prefixes by default. Route propagation is enabled per route table:

resource "aws_vpn_gateway_route_propagation" "private" {
  for_each = aws_route_table.private
 
  vpn_gateway_id = aws_vpn_gateway.main.id
  route_table_id = each.value.id
}

Miss one route table and the subnets associated with it cannot reach on-premises, while every other subnet can. That produces the same shape of incident as Lab 03: identical-looking subnets where some work and some do not, because propagation is a per-route-table relationship rather than a property of the connection.

Iterate over the route tables rather than listing them. A hand-maintained list is what lets the next subnet be added without its propagation.

Route priority, and the rule that surprises people

Within a VPC route table, priority runs (VPC docs):

Longest prefix match
Static routes
Prefix list routes
Propagated routes, themselves ordered: Direct Connect BGP → VPN static → VPN BGP

Two consequences worth committing to memory.

A static route beats a propagated route at the same prefix. If on-premises advertises 172.31.0.0/24 over the VPN and someone has a static 172.31.0.0/24 route pointing at an internet gateway, the static route wins and your on-premises traffic goes to the internet.

rtb-privateassoc: subnet-private-1a1 route not forwarding

Destination	Target	State	Origin
10.0.0.0/16	local	activeThe local route always wins for this range — see below.	CreateRouteTable
172.31.0.0/24	igw-04c…	activeStatic beats propagated at the same prefix, so this is the route that forwards.	static
172.31.0.0/24	vgw-11a…	inactiveLearned over BGP from on-premises and shadowed by the static route above.	propagated

Both 172.31.0.0/24 entries are present and healthy. Priority, not state, decides which one forwards — so nothing here reports an error.

The local route always wins, even against a more specific propagated route. If on-premises advertises a prefix that overlaps your VPC CIDR, the VPC's local route takes precedence regardless (VPN route priority). You cannot reach an on-premises host whose address falls inside your own VPC range. This is the concrete reason overlapping address space is unfixable rather than merely inconvenient — no route you can write overrides local.

Transit Gateway evaluates differently

A Transit Gateway orders by attachment type rather than the VPC list (TGW docs): most specific prefix first, then static → prefix list → VPC-propagated → Direct Connect gateway-propagated → Transit Gateway Connect → VPN over private Direct Connect → VPN. For routes with the same CIDR and attachment type, BGP attributes decide: shorter AS_PATH, then lower MED, then eBGP over iBGP.

Also worth knowing for failover design: when a route arrives without MED, Transit Gateway assigns a default of 0 for Direct Connect attachments and 100 for VPN and Connect attachments. Lower MED wins, so Direct Connect is preferred over VPN by default without you configuring anything.

Influencing which path AWS chooses

Everything above governs traffic leaving AWS. Controlling that direction is what people mean by "influencing inbound traffic to on-premises," and Direct Connect evaluates in this order (routing policies):

Longest prefix length
Local preference, set with BGP community tags — 7224:7100 low, 7224:7200 medium, 7224:7300 high
AS_PATH length
MED — works, but AWS does not recommend relying on it given its low priority
ECMP across multiple private or transit VIFs when AS_PATH length and other attributes match

So for active/passive: advertise a more specific prefix on the active link, or tag it 7224:7300 and the standby 7224:7100. For active/active: advertise identical prefixes with identical attributes and let ECMP distribute.

Asymmetric routing is the recurring failure

Hybrid designs make it easy to have traffic leave over one path and return over another. IP routing is stateless, so asymmetry is not inherently a problem — until something stateful sits in the path.

10.0.11.20 → 192.168.40.15:443, request over Direct Connect, reply over VPN

EC2
Request leaves the VPC
route table prefers the Direct Connect propagated route
GW
Direct Connect → on-premises firewall A
firewall A creates state for this flow
DEST
On-premises host answers
reply is routed by the corporate network, not by AWS
GW
Reply egresses via the VPN, through firewall B
corporate routing prefers the VPN for this return prefix
Dropped — firewall B has no state for a flow it never saw the start of, so it drops the reply
DEST
Application host
sees a connection that established and then stalled

Both paths are healthy and both firewalls are behaving correctly. The defect is that one flow traversed two of them.

The signature is distinctive: connections that establish and then hang, or that work for short requests and fail for long ones, correlating with nothing obvious. Symmetry is something you have to design for — matching prefix specificity and BGP attributes in both directions so a flow's two halves take the same path.

ECMP makes this sharper. Transit Gateway ECMP hashes on a 5-tuple across tunnels, which AWS notes produces asymmetric routing by design — fine for stateless IP, and requiring a source NAT rule if your customer gateway is a stateful firewall.

Bandwidth, concretely

A single Site-to-Site VPN tunnel caps at roughly 1.25 Gbps, and that is a per-flow ceiling you cannot exceed by adding tunnels. ECMP across two tunnels on a Transit Gateway reaches about 2.5 Gbps aggregate, and more connections scale further — but any individual flow is still limited to ~1.25 Gbps because the 5-tuple hash pins it to one tunnel.

Practical consequence: if a single large transfer is slow, more tunnels will not help. That is the point at which the answer is Direct Connect, not more VPN.

ECMP requires a Transit Gateway with dynamic routing enabled. It is not supported on a VGW at all — another way the endpoint choice constrains you later.

Redundancy that actually is redundant

Every Site-to-Site VPN connection comes with two tunnels to two separate AWS endpoints. That is not optional and it is not the same as redundancy: both tunnels terminate on your one customer gateway. If that device or its site fails, both go down together.

Real redundancy means a second customer gateway, ideally at a second site, and for Direct Connect it means connections at multiple locations terminating on different routers. A common and reasonable posture is Direct Connect primary with a VPN backup on a Transit Gateway, which the default MED values already prefer correctly.

What to instrument

TunnelState per tunnel. Alarm on either dropping, not just both — losing one tunnel is a silent loss of redundancy that stays invisible until the second one goes.
BGP session state. Given that exceeding the route ceiling moves the session to Idle, an alarm on session state combined with a check on advertised route count catches the 100-route failure before someone notices the outage.
Advertised route count, trended. This grows through other teams' changes, so a threshold alert at 80 of 100 buys you warning rather than an incident.
PacketDropCountNoRoute on the Transit Gateway, if you terminate there.

Where this leaves you

The through-line across this track and the others: AWS networking failures are usually about a relationship between two objects rather than one broken object. A VGW that needs propagation enabling per route table. A static route shadowing a propagated one. A flow whose two halves take different paths through different firewalls. In every case each object is healthy and the pairing is wrong.

There is no hands-on lab for this track yet — Site-to-Site VPN needs a simulated customer gateway, and a lab that has not been applied against a real account is not one worth shipping. In the meantime, Lab 03 drills the propagation model that the VGW section above depends on.