Hybrid Connectivity: VPN, Direct Connect, and the BGP Attributes That Actually Work

Hybrid connectivity looks like two decisions — VPN or Direct Connect, and which prefixes to advertise. In practice the consequential choice is a third one nobody frames as a decision: whether you terminate on a virtual private gateway or a Transit Gateway. That single pick sets your route quotas, determines whether ECMP is available at all, and changes how failover behaves. Get it wrong and you discover the constraint during a migration, at which point moving is a project.

The AWS-side endpoint decides your ceilings#

Both a VGW and a Transit Gateway will terminate a Site-to-Site VPN. They are not interchangeable.

Quotas per the Site-to-Site VPN quotas page; none of them are adjustable.

The 100-route figure is the one that causes outages, because of how it fails.

This is a bad failure mode for a specific reason: the trigger is usually somewhere else entirely. Nobody changes the VPN. Someone adds a subnet on the corporate network, a redistribution rule picks it up, the advertised count crosses 100, and a link that has been stable for two years starts flapping. The change and the symptom are in different teams' systems.

Mitigations, in descending order of preference: summarize on the customer gateway so you advertise a handful of aggregates instead of every subnet; advertise a default route to AWS if your topology allows it; or terminate on a Transit Gateway, where the ceiling is 1,000.

Direct Connect has the same failure, with a number you choose#

Terminating on a Transit Gateway raises the VPN ceiling, and it is easy to read that as "the prefix problem is solved." It is not. Direct Connect has its own inbound ceiling, and it fails identically.

A word on vocabulary before the numbers, because the rest of this section leans on it. A virtual interface is one VLAN on a Direct Connect connection carrying its own BGP session, and it comes in three types: private reaches VPCs over private addresses, transit reaches Transit Gateways through a Direct Connect gateway, and public reaches AWS public service endpoints and involves no gateway at all. Only the first two have prefix allocations, which is why the public type does not appear again below. The Direct Connect reference sheet has the full table of what each type terminates on.

Direct Connect manages this through inbound prefix controls, which allocate a prefix budget to each private or transit virtual interface out of two pools — one on the dedicated connection, one on the Direct Connect gateway.

Two things about that table matter more than the numbers.

The first is the default. 100 per address family, per VIF — the same figure as the VGW ceiling, and the same consequence: advertise more than your allocation and the BGP session on that VIF goes down and enters an idle state. Not the excess prefixes. The session. If you provisioned a transit VIF and never thought about prefix allocation, you are running on 100, and the trigger is once again a change on somebody else's network.

The second is that it is now a number you set deliberately, up to 1,000 per VIF. That makes it a provisioning decision rather than a fixed constraint — which is better, but only if someone makes it. AWS also notes you cannot reduce an allocation below what is currently in use, so the fix direction is one-way in a hurry.

Worth knowing when you plan a Direct Connect gateway: the 10,000 total is shared across all attached VIFs, so ten transit VIFs at the 1,000 maximum exhausts it exactly. An AWS Interconnect connection consumes 2,000 of that budget on its own.

Propagation is opt-in on a VGW too#

If you terminate on a VGW, the VPC route tables do not learn your on-premises prefixes by default. Route propagation is enabled per route table:

resource "aws_vpn_gateway_route_propagation" "private" {
  for_each = aws_route_table.private
 
  vpn_gateway_id = aws_vpn_gateway.main.id
  route_table_id = each.value.id
}

Miss one route table and the subnets associated with it cannot reach on-premises, while every other subnet can. That produces the same shape of incident as Lab 03: identical-looking subnets where some work and some do not, because propagation is a per-route-table relationship rather than a property of the connection.

Iterate over the route tables rather than listing them. A hand-maintained list is what lets the next subnet be added without its propagation.

Route priority, and the rule that surprises people#

Within a VPC route table, priority runs (VPC docs):

1. Longest prefix match
2. Static routes
3. Prefix list routes
4. Propagated routes, themselves ordered: Direct Connect BGP → VPN static → VPN BGP

Two consequences worth committing to memory.

A static route beats a propagated route at the same prefix. If on-premises advertises 172.31.0.0/24 over the VPN and someone has a static 172.31.0.0/24 route pointing at an internet gateway, the static route wins and your on-premises traffic goes to the internet.

The local route always wins, even against a more specific propagated route. If on-premises advertises a prefix that overlaps your VPC CIDR, the VPC's local route takes precedence regardless (VPN route priority). You cannot reach an on-premises host whose address falls inside your own VPC range. This is the concrete reason overlapping address space is unfixable rather than merely inconvenient — no route you can write overrides local.

Transit Gateway evaluates differently#

A Transit Gateway orders by attachment type rather than the VPC list (TGW docs): most specific prefix first, then static → prefix list → VPC-propagated → Direct Connect gateway-propagated → Transit Gateway Connect → VPN over private Direct Connect → VPN. For routes with the same CIDR and attachment type, BGP attributes decide: shorter AS_PATH, then lower MED, then eBGP over iBGP.

Also worth knowing for failover design: when a route arrives without MED, Transit Gateway assigns a default of 0 for Direct Connect attachments and 100 for VPN and Connect attachments. Lower MED wins, so Direct Connect is preferred over VPN by default without you configuring anything.

Influencing which path AWS chooses#

Everything above governs traffic leaving AWS. Controlling that direction is what people mean by "influencing inbound traffic to on-premises," and Direct Connect evaluates in this order (routing policies):

1. Longest prefix length
2. Local preference, set with BGP community tags — 7224:7100 low, 7224:7200 medium, 7224:7300 high
3. AS_PATH length
4. MED — works, but AWS does not recommend relying on it given its low priority
5. ECMP across multiple private or transit VIFs when AS_PATH length and other attributes match

So for active/passive: advertise a more specific prefix on the active link, or tag it 7224:7300 and the standby 7224:7100. For active/active: advertise identical prefixes with identical attributes and let ECMP distribute.

Asymmetric routing is the recurring failure#

Hybrid designs make it easy to have traffic leave over one path and return over another. IP routing is stateless, so asymmetry is not inherently a problem — until something stateful sits in the path.

The signature is distinctive: connections that establish and then hang, or that work for short requests and fail for long ones, correlating with nothing obvious. Symmetry is something you have to design for — matching prefix specificity and BGP attributes in both directions so a flow's two halves take the same path.

ECMP makes this sharper. Transit Gateway ECMP hashes on a 5-tuple across tunnels, which AWS notes produces asymmetric routing by design — fine for stateless IP, and requiring a source NAT rule if your customer gateway is a stateful firewall.

Bandwidth, concretely#

A standard Site-to-Site VPN tunnel caps at roughly 1.25 Gbps, and that is a per-flow ceiling you cannot exceed by adding tunnels. ECMP across two tunnels on a Transit Gateway reaches about 2.5 Gbps aggregate, and more connections scale further — but any individual flow is still limited to ~1.25 Gbps because the 5-tuple hash pins it to one tunnel.

So if a single large transfer is slow, more tunnels will not help. The instinct at that point is to reach for Direct Connect, and there is a step in between: a Large Bandwidth Tunnel raises the per-tunnel ceiling to 5 Gbps, which is a per-flow improvement rather than an aggregate one. AWS's framing is that it "eliminates or reduces the need to deploy complex protocols such as ECMP", and lists backing a 10 Gbps-plus Direct Connect circuit as an intended use. Above 5 Gbps you combine the two — AWS's own example reaches 20 Gbps with two Large Bandwidth connections and ECMP across all four tunnels.

The constraints are worth reading before you plan around it: both tunnels on a connection must use the same bandwidth setting, Accelerated VPN is not supported, the customer gateway must have a fixed IP, and it is unavailable in five Regions.

Note which way that cuts. Large Bandwidth Tunnels are available only on VPN connections attached to a Transit Gateway or Cloud WAN — not on a virtual private gateway. So does ECMP, which needs a Transit Gateway with dynamic routing enabled and is not supported on a VGW at all. The endpoint choice from the top of this guide turns out to set your bandwidth ceiling as well as your route ceiling, which is the strongest argument in the piece for not terminating on a VGW.

Redundancy that actually is redundant#

Every Site-to-Site VPN connection comes with two tunnels to two separate AWS endpoints. That is not optional and it is not the same as redundancy: both tunnels terminate on your one customer gateway. If that device or its site fails, both go down together.

Real redundancy means a second customer gateway, ideally at a second site, and for Direct Connect it means connections at multiple locations terminating on different routers. A common and reasonable posture is Direct Connect primary with a VPN backup on a Transit Gateway, which the default MED values already prefer correctly.

How long failover actually takes#

Having a second path is not the same as failing over to it quickly, and this is the question that gets asked in interviews and skipped in designs.

By default, the answer is BGP's hold timer. AWS's BGP settings default to a 90-second hold timer with a 30-second keepalive, so a path that fails without the session cleanly tearing down — a link that goes quiet rather than down, which is the common case for anything crossing a provider — leaves you blackholing traffic for up to a minute and a half. The route is still in the table. It is simply wrong.

Bidirectional Forwarding Detection is the fix, and it is a small protocol whose only job is to notice quickly. AWS supports a 300 ms minimum liveness interval with a minimum multiplier of 3, so detection lands near 900 ms — roughly two orders of magnitude faster than the hold timer.

Three details that decide whether you get that in practice:

• The slower peer wins. BFD intervals negotiate to the maximum of your desired transmit interval and the peer's minimum receive interval, so configuring 300 ms locally means nothing if the other end asks for 1 second. AWS also negotiates each direction independently, which means your BFD intervals can legitimately be asymmetric.
• Detection time is negotiated interval × local multiplier. Both halves are per-device, so two ends of the same session can genuinely disagree about how long a failure takes to notice.
• Do not run it alongside graceful restart. AWS recommends against configuring graceful restart and BFD together, and the reason is that they want opposite things: BFD exists to tear a session down fast, graceful restart exists to preserve forwarding across a restart. Enable both and the mechanism meant to speed up failover is arguing with the mechanism meant to suppress it.

Note also that Direct Connect sets a BGP packet TTL of 1 and does not support multihop BGP, so terminating the session on anything other than the device at the end of the circuit is not an option.

What to instrument#

• TunnelState per tunnel. Alarm on either dropping, not just both — losing one tunnel is a silent loss of redundancy that stays invisible until the second one goes.
• BGP session state. Given that exceeding the route ceiling moves the session to Idle, an alarm on session state combined with a check on advertised route count catches the 100-route failure before someone notices the outage.
• Advertised route count, trended. This grows through other teams' changes, so a threshold alert at 80 of 100 buys you warning rather than an incident. You may not have to build the trend yourself: Site-to-Site VPN logs carry a maximum-prefix warning before the session dies — received 76 prefixes from neighbor …, limit is 100 — followed by a BGP Cease/Maximum Number of Prefixes Reached notification when it does. Configuring maximum-prefix with warning-only on the customer gateway gives you the same signal from your own side.
• PacketDropCountNoRoute on the Transit Gateway, if you terminate there.
• Prefix allocation headroom on each Direct Connect VIF, for the same reason as the advertised route count above. The default allocation is 100 and the failure is a downed BGP session, so this wants the same 80-percent warning rather than a discovery during an incident.

Where this leaves you#

The through-line across this track and the others: AWS networking failures are usually about a relationship between two objects rather than one broken object. A VGW that needs propagation enabling per route table. A static route shadowing a propagated one. A flow whose two halves take different paths through different firewalls. In every case each object is healthy and the pairing is wrong.

Where to practice#

Lab 07 is this track's failure, end to end: a Site-to-Site VPN whose connection is available, whose tunnel is UP, and whose static route for the branch prefix is listed on the connection — carrying nothing, because propagation was never enabled on the route table the subnet actually consults. It builds a simulated customer gateway so there is a real tunnel, and it is deliberately structured so the diagnosis works from the AWS API even if that tunnel does not come up.

Lab 03 is the Transit Gateway counterpart, and worth doing alongside it. Same word, different resource, same class of defect — which is the point of putting them next to each other.