Direct Connect has a button that breaks it on purpose, and almost nobody presses it

Every hybrid design has a failover story. The Direct Connect goes down, BGP withdraws the routes, the backup VPN takes over, nobody notices. The story is in the architecture diagram and the runbook and the review that approved it.

The number of estates where that story has been observed to happen, rather than reasoned to happen, is smaller than it should be. Not because testing is hard. Because testing means breaking a production circuit, and nobody wants to be the one who did that.

AWS built a way to do it that is not that.

The API#

From the Direct Connect API reference:

Starts the virtual interface failover test that verifies your configuration meets your resiliency requirements by placing the BGP peering session in the DOWN state. You can then send traffic to verify that there are no outages. You can run the test on public, private, transit, and hosted virtual interfaces.

It takes one BGP peering session — the IPv4 or IPv6 peer on one virtual interface — and puts it in DOWN for a duration you specify. The physical connection stays up. The other peerings on the same connection stay up. Only the routes that session was advertising are withdrawn, which is exactly what a real failure of that path would do from the routing layer's point of view.

aws directconnect start-bgp-failover-test \
  --virtual-interface-id dxvif-0123456789abcdef0 \
  --test-duration-in-minutes 20

And in the console it is Actions → Bring down BGP on the virtual interface, which is the most direct label AWS has ever put on a button.

Why this is the only test that counts#

The hybrid connectivity guide makes a point about Transit Gateway route tables that is the whole argument for this feature: a route table shows only the preferred route. If Direct Connect and a backup VPN advertise the same prefixes, the VPN's routes are not in the table at all until Direct Connect stops advertising them. The guide's instruction is "verify failover by withdrawing the primary, not by reading the table."

This API is the withdrawal. Reading the route table before the test tells you Direct Connect is preferred, which you knew. Reading it during the test tells you whether the VPN routes actually appeared, whether traffic actually moved, and how long the gap was — which is the thing you did not know and could not learn any other way short of an outage.

What it exercises, in order:

1. BGP on your side notices. Your router's hold timer, or BFD if you have it, detects the dead peer. This is where the BFD negotiation decides whether detection takes a second or ninety.
2. The backup path's routes are installed. On a Transit Gateway, VPN routes that were hidden behind the Direct Connect ones become the active routes. On a VGW, the propagated routes change source.
3. Traffic actually flows over the backup. Which tests the VPN's bandwidth, its security groups, its MTU, and every other thing that was never exercised because the VPN never carried anything.
4. Return traffic comes back the same way. The asymmetric-routing failure — out via VPN, back via a path that no longer exists — shows up here and nowhere else.

A failover test that does not run end to end does not test step three or four, and those are the two that fail in practice.

The duration default will surprise you#

Two numbers from the same API page:

testDurationInMinutes — The time in minutes that the virtual interface failover test will last. Maximum value: 4,320 minutes (72 hours). Default: 180 minutes (3 hours).

Three hours, if you do not specify. That is a long time to have a production path down if you were expecting a five-minute check, and the console dialog defaults to the same 180. Set the duration deliberately, and set it short for a first run — twenty minutes is enough to watch convergence, run traffic, and read every metric you care about.

The upper bound of 72 hours exists for a different kind of test: proving the backup path can carry production for a sustained period, not just converge to it. That is a test worth running once, with a change window, before you believe the backup is a backup.

It can be stopped#

The reason this is safe to run against production:

If you need to stop the test before the test interval completes, use StopBgpFailoverTest.

aws directconnect stop-bgp-failover-test \
  --virtual-interface-id dxvif-0123456789abcdef0

The resiliency troubleshooting article states it without qualification: "You can stop the failover test at any time." If the backup does not take over, you stop the test and the BGP session comes back. You have learned that your failover does not work, at the cost of however many seconds it took you to notice and press stop, instead of at the cost of an outage of unknown length at a time nobody chose.

That asymmetry is the whole case. The test fails safe. The outage does not.

It leaves a record#

You can use ListVirtualInterfaceTestHistory to view the virtual interface test history.

aws directconnect list-virtual-interface-test-history \
  --virtual-interface-id dxvif-0123456789abcdef0 \
  --query 'virtualInterfaceTestHistory[].[testId,startTime,endTime,testDurationInMinutes,status]' \
  --output table

This is the artefact a resilience review should ask for. Not the diagram — the test history, with dates, for every virtual interface that has a failover story attached to it. A virtual interface with an empty history has a failover story that has never been told to the network.

Two constraints#

Only the account that owns the virtual interface can start the test. For a hosted virtual interface — one allocated to your account by a partner or another account — the test runs against your interface, but the article notes "only the owner of the AWS account that includes the virtual interface can initiate the test." In a multi-account estate, that is the account holding the VIF, not the one holding the connection.

It tests one peering at a time. bgpPeers selects IPv4 or IPv6 on one virtual interface. A connection with two VIFs needs two tests to prove both fail over, and a connection with dual-stack peering needs the IPv6 peer tested separately — which is worth doing, because IPv6 fails independently of IPv4 and a failover design that was only ever tested on IPv4 has only ever been tested on half the traffic.

What to run alongside it#

The test takes BGP down. It does not tell you what happened. Three things to have open before pressing the button:

The Transit Gateway route table, if that is where the VIF terminates, polled through the test:

watch -n 5 "aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id $RTB \
  --filters Name=state,Values=active \
  --query 'Routes[].[DestinationCidrBlock,TransitGatewayAttachments[0].ResourceType,State]' --output table"

You are watching for the direct-connect-gateway rows to be replaced by vpn rows, and timing how long it takes.

A continuous flow from an instance to an on-premises host, with timestamps, so the gap is measured rather than estimated:

ping -D -i 1 10.50.0.10 | tee failover-$(date +%s).log

The VPN's TunnelDataIn and TunnelDataOut metrics in CloudWatch. If the backup is carrying traffic, these move. If they stay flat during the test, traffic went somewhere else or nowhere.

The organisational point#

A failover design that has not been exercised is a hypothesis. Most teams keep it a hypothesis because the only way to test it was to cause the thing it protects against. AWS removed that excuse with an API that does the withdrawal reversibly, logs that it happened, and lets you stop it the moment it goes wrong.

The remaining barrier is a change ticket and twenty minutes. Against the alternative — finding out during a fibre cut — that is not a hard trade. Press the button. Set the duration first.