Lab 24: The Gateway Load Balancer Endpoint That Took Traffic In And Never Let It Out
An inspection path built from the getting-started guide. The appliance is healthy, the endpoint is available, and every packet the application sends is inspected and handed back. None of it reaches the internet, and the appliance is the one place in the path that can prove it did its job.
- Debugging time
- ~35 min
- Reading time
- 13 min
- Reported by
- Security Engineering
- Tier
- Professional
Application subnet lost all internet access after inspection cutover; appliance healthy, endpoint available
Reported by Security Engineering
We cut the application subnet over to inline inspection last night: a Gateway Load Balancer in the security VPC, a Gateway Load Balancer endpoint in the application VPC, ingress routing on the internet gateway, default route on the application subnet pointed at the endpoint. Built from the AWS getting-started guide.
Since the cutover the application host cannot reach anything on the internet, and nothing from the internet completes to it. Package updates hang, the outbound webhook is timing out, the health checker from outside is red.
The parts I can check all check out. The endpoint is available. The appliance is healthy in the
target group. The appliance team ran a capture and say they are receiving our traffic and sending it
back — they are adamant the problem is not theirs. Security groups and network ACLs have not been
touched. There is no error anywhere. The traffic just goes in and does not come out.
What you are working with
Two VPCs in one Availability Zone. The provider side holds the Gateway Load Balancer and one appliance. The consumer side holds the application host and the endpoint.
Ingress routing sends internet traffic for the application subnet to the endpoint before it reaches the host. The application subnet's default route sends everything outbound to the same endpoint. Both of those are correct. The endpoint subnet's own route table is the one thing this diagram leaves blank, because it is the thing you are going to read. The SSM endpoints are scaffolding: they are why you can reach the application host at all while it is cut off.
| Resource | Configuration |
|---|---|
| Gateway Load Balancer | Provider VPC, one zone, listener → GENEVE target group |
| Target group | Protocol GENEVE, port 6081, health check TCP:22 |
| Appliance | t3.micro, registered, healthy; hairpins every GENEVE frame back |
| Endpoint service | Gateway type, acceptance not required |
| GWLB endpoint | Consumer VPC, subnet 10.191.0.0/24, available |
| Internet gateway route table | 10.191.1.0/24 → vpce (edge association) |
| Application subnet route table | 0.0.0.0/0 → vpce |
| Endpoint subnet route table | read it |
| Application host | 10.191.1.50, public IP, Python HTTP server on :80 |
- EC2
App host
10.191.1.50 → internet
- RTB
Application subnet route table
0.0.0.0/0 → vpce
- VPCE
GWLB endpoint
PrivateLink to the provider VPC
- GW
Gateway Load Balancer
GENEVE encapsulation, UDP 6081
- VPCE
Appliance
frame received, frame returned
- GW
Gateway Load Balancer
decapsulates, returns to the endpoint
- RTB
Endpoint subnet route table
destination: the internet
Dropped — The only route in this table is the local one. The packet has been inspected and emitted into the endpoint subnet, and there is no route from here to the internet gateway. The VPC router has nothing to match and discards it.
- DEST
the internet
never reached
Six hops succeed. The seventh is a route table with one entry in it. Everything the ticket checked is upstream of the failure, and everything upstream is working.
Scope and constraints
- In scope: why traffic that is inspected and returned never reaches its destination.
- Out of scope: the load balancer, the target group, the health check, the endpoint service, the endpoint, the ingress routing association, the application subnet's route, security groups and network ACLs. All correct. The reproduction confirms each so you can stop re-checking them.
- The appliance team is right. Take their claim seriously and verify it on their host, because once you believe it the search space collapses.
- The fix adds one route to one table. Nothing is removed and nothing is reconfigured.
Deploy the broken state
cd lab-24-gwlb-endpoint-return-route
terraform init
terraform applyAllow about six minutes. The Gateway Load Balancer takes a few to become active, the endpoint a
little longer to become available, and the appliance installs its hairpin on first boot after it
has found the load balancer's address. Before testing, confirm the target is healthy:
aws elbv2 describe-target-health \
--target-group-arn "$(terraform output -raw target_group_arn)" \
--query 'TargetHealthDescriptions[].TargetHealth.State' --output text
# healthyIf it reports initial, wait a minute and run it again.
APP=$(terraform output -raw app_instance_id)
APP_IP=$(terraform output -raw app_public_ip)
GWLBE=$(terraform output -raw gwlb_endpoint_id)
RTB=$(terraform output -raw gwlbe_route_table_id)Confirm what the ticket confirmed
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids "$GWLBE" \
--query 'VpcEndpoints[0].[VpcEndpointType,State]' --output text
# GatewayLoadBalancer available
aws elbv2 describe-target-health \
--target-group-arn "$(terraform output -raw target_group_arn)" \
--query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State]' --output text
# i-0a1b… healthyBoth true. Note what healthy means here: the health check is a plain TCP connection to port 22 on
the appliance. It is not GENEVE-encapsulated and it never traverses the endpoint. A healthy target
tells you the load balancer can open a TCP connection to the appliance, and nothing else.
Reproduce the outage, both directions
Open a session on the application host:
terraform output -raw start_session_commandOutbound:
curl -s -m 5 https://checkip.amazonaws.com; echo "exit $?"
# exit 28Exit 28 is curl's timeout. DNS worked — the resolver is reached over the local route — and then the connection went nowhere.
Inbound, from your own machine against the host's public address:
curl -s -m 5 "http://$APP_IP/"; echo "exit $?"
# exit 28While that is retrying, back in the session on the application host:
ss -tn state syn-recv
# Recv-Q Send-Q Local Address:Port Peer Address:Port
# 0 0 10.191.1.50:80 203.0.113.7:51844That line is the finding for the inbound direction. SYN-RECV means your SYN arrived at the host and
the host answered with a SYN-ACK that never got an ACK back. The inbound path through the endpoint
delivered. The outbound path did not.
Verify the appliance team's claim
Open a second session, on the appliance:
terraform output -raw appliance_session_commandStart a capture of GENEVE traffic, then from the application host's session run the outbound
curl again:
sudo tcpdump -ni any -c 6 udp port 6081
# 10.190.1.183.6081 > 10.190.1.21.6081: Geneve, ...
# 10.190.1.21.6081 > 10.190.1.183.6081: Geneve, ...
# 10.190.1.183.6081 > 10.190.1.21.6081: Geneve, ...
# 10.190.1.21.6081 > 10.190.1.183.6081: Geneve, ...Pairs. For every frame that arrives from the load balancer, one leaves for it. The hairpin counters say the same:
sudo iptables -t nat -L PREROUTING -v -n | grep DNAT
# 6 ... udp -- 10.190.1.183 10.190.1.21 to:10.190.1.183:6081The appliance team is right. Every packet the application sends reaches the appliance and is returned to the load balancer. The failure is after that.
Read the third route table
The getting-started guide configures three route tables in the consumer VPC. Read all three.
The internet gateway's, found by its edge association:
aws ec2 describe-route-tables \
--filters "Name=association.gateway-id,Values=$(terraform output -raw consumer_internet_gateway_id)" \
--query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,State]' --output table
# -----------------------------------------------
# | 10.191.0.0/16 | local | active |
# | 10.191.1.0/24 | vpce-0f3a… | active |
# -----------------------------------------------The application subnet's:
aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=$(aws ec2 describe-instances --instance-ids "$APP" \
--query 'Reservations[0].Instances[0].SubnetId' --output text)" \
--query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,State]' --output table
# -----------------------------------------------
# | 10.191.0.0/16 | local | active |
# | 0.0.0.0/0 | vpce-0f3a… | active |
# -----------------------------------------------The endpoint subnet's:
aws ec2 describe-route-tables --route-table-ids "$RTB" \
--query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,State]' --output table
# -----------------------------------------------
# | 10.191.0.0/16 | local | active |
# -----------------------------------------------One route. The packet the appliance just returned is emitted into this subnet with a destination on the internet, and this is the table the VPC router consults to deliver it.
This debrief is part of Labs Pro
The root-cause analysis, packet-flow walkthrough, and Terraform remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The brief, the reproduction steps, and the Terraform stay free — you can still solve this one yourself.
Already bought it? Sign in and it unlocks.