Lab 17: The Appliance Mode That Was Enabled And Ignored

What you are working with#

One Transit Gateway, three VPCs, two Availability Zones.

Source and destination are in the same Availability Zone. Per AWS's documented Scenario 1, appliance mode should hold that zone for the whole flow, so every flow should reach appliance A.

Scope and constraints#

• In scope: why zone affinity is not happening when both documented requirements are met.
• Out of scope: appliance mode itself, the propagation configuration, the attachment subnets, and the spoke route table. All correct.
• The reporter is right on both counts. Appliance mode is enabled and propagation is enabled. Verify both yourself early so you stop re-checking them.
• Nothing is broken. No connection fails and no alarm fires. This is a cost and blast-radius finding.
• The fix removes configuration rather than adding it.

Deploy the broken state#

cd lab-17-appliance-mode-static-shadow
terraform init
terraform apply

The Transit Gateway takes a few minutes and the attachments a few more — budget about eight minutes before anything is testable.

CLIENT=$(terraform output -raw client_instance_id)
APP_A=$(terraform output -raw appliance_a_instance_id)
APP_B=$(terraform output -raw appliance_b_instance_id)
RTB=$(terraform output -raw appliance_route_table_id)

Confirm both documented requirements are met#

Do this first, because it is what the ticket claims and it is true.

aws ec2 describe-transit-gateway-vpc-attachments \
  --transit-gateway-attachment-ids "$(terraform output -raw appliance_attachment_id)" \
  --query 'TransitGatewayVpcAttachments[0].[State,Options.ApplianceModeSupport]' --output text
# available	enable
 
aws ec2 get-transit-gateway-route-table-propagations \
  --transit-gateway-route-table-id "$RTB" \
  --query 'TransitGatewayRouteTablePropagations[].[TransitGatewayAttachmentId,State]' --output table
# ----------------------------------------------------
# |  tgw-attach-0616db2cd42ac0fe4 |  vpc |  enabled  |
# |  tgw-attach-0edecca4c33d992c6 |  vpc |  enabled  |
# ----------------------------------------------------

Appliance mode enabled. Propagation enabled on both spoke attachments. Both boxes ticked.

Measure, do not test#

A single connection tells you nothing here, because each individual flow succeeds and is handled by exactly one appliance. You have to count.

Start a capture on each appliance. These run in the foreground and their output is the result, so launch both and leave them:

# On appliance A, and separately on appliance B
sudo timeout 80 tcpdump -ni any -nn -c 500 "tcp and dst host 10.181.1.50"

Then, from the client, open sixty distinct flows. Varying the destination port varies the tuple, and therefore the hash:

for p in $(seq 4000 4059); do
  timeout 1 bash -c "echo > /dev/tcp/10.181.1.50/$p" 2>/dev/null
done
echo done

Count the distinct destination ports each appliance saw:

# piped from each capture
sed -n 's/.*> 10\.181\.1\.50\.\([0-9]*\):.*/\1/p' | sort -un | wc -l

Measured on a real account:

appliance A (first AZ)   24 flows
appliance B (second AZ)  29 flows

Both endpoints are in the first Availability Zone. Appliance B should have seen nothing. It saw more than half.

Note what is working#

Two things worth registering before you go looking for the cause, because they narrow it sharply.

The port sets are disjoint. No flow appears on both appliances. Each flow is pinned to one appliance for its lifetime, which is appliance mode's flow stickiness working correctly. Whatever is broken is not the stickiness.

The split is close to even. That is the signature of a hash over a roughly uniform input, not of a partial failure or a health problem. Something is choosing between two zones at random rather than being told which zone to use.

So the failure is specifically in zone determination, and appliance mode is otherwise functioning.

Look at what the route table actually contains#

Propagation is enabled. Look at the routes themselves rather than the propagation state:

aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id "$RTB" \
  --filters "Name=state,Values=active" \
  --query 'Routes[].[DestinationCidrBlock,Type,State]' --output table
# ---------------------------------------
# |  10.180.0.0/16|  static  |  active  |
# |  10.181.0.0/16|  static  |  active  |
# ---------------------------------------

static.

Propagation is enabled and there is not a single propagated route in the table. Two static routes for the same prefixes are taking precedence, so the propagated routes exist as configuration and are never the routes in use.