Lab 27: The Lattice Target That Would Only Talk To The Client
An application whose security group admits HTTP from the client security group, which is how everything else in the VPC is wired and which works. Put a VPC Lattice service in front of it and the same client gets nothing, the target reports unhealthy, and the rule that was supposed to admit the client was never going to see the client.
- Debugging time
- ~30 min
- Reading time
- 10 min
- Reported by
- Platform Engineering
- Tier
- Professional
New VPC Lattice service returns errors; target unhealthy; same client reaches the application directly
Reported by Platform Engineering
We are piloting VPC Lattice for service-to-service calls. First service: the inventory API, one EC2
target, HTTP listener on 80, the VPC associated with the service network. Auth type is NONE for
the pilot so nothing about IAM should be in the way.
The client cannot get a response through the service DNS name. The target shows UNHEALTHY in the
target group. But the application is fine: from the same client instance, curl straight at the
application's private IP returns 200 immediately.
The application's security group allows port 80 from the client's security group. That rule is how every service in this VPC is secured and it has worked for two years. We have not changed it.
What you are working with
One VPC, one subnet, two instances, one VPC Lattice service.
The direct path from client to application is a packet whose source carries the client's security group, and sg-app admits it. The path through the Lattice service leaves the VPC's addressing entirely: the client talks to a link-local address, and the application receives a connection from a VPC Lattice address. sg-app has no rule for that, and the health checks arrive the same way.
| Resource | Configuration |
|---|---|
| Client | t3.micro, sg-client, egress all |
| App | t3.micro, 10.27.1.50, Python HTTP server on :80 |
sg-app inbound | TCP 80 from sg-client, nothing else |
| Service network | auth_type = NONE, VPC associated, no association security groups |
| Service | auth_type = NONE, HTTP listener on 80 → target group |
| Target group | INSTANCE, HTTP:80, health check GET / expecting 200 |
- EC2
Client
curl http://<service dns>/
- RTB
DNS
service name resolves to a link-local address
- VPCE
VPC Lattice
terminates the client connection, opens one to the target
- FILTER
sg-app
inbound: TCP 80 from sg-client
Dropped — The connection to the target comes from a VPC Lattice address, not from an interface in sg-client. The only inbound rule names sg-client. Nothing matches, and the security group denies by default. Health checks take the same path and meet the same rule.
- DEST
App
never reached
The client is not the thing connecting to the application. VPC Lattice is. A rule that admits the client admits nothing on this path.
Scope and constraints
- In scope: why a security group rule that works for direct traffic admits nothing through VPC Lattice, and what the source of Lattice traffic is.
- Out of scope: auth policies (auth is
NONEthroughout), the service network association's own security groups, the listener, the target group, DNS. All correct. - The reporter is right that the application works directly and right that the rule has not changed. Both are evidence.
- The fix is one inbound rule, and it is one that AWS's own documentation says is the only kind that can work here.
Deploy the broken state
cd lab-27-lattice-target-security-group
terraform init
terraform applyAbout five minutes. The service, the target group attachment and the two associations each take a minute or two to become active. Wait for the target group to finish its initial health checks before reproducing:
aws vpc-lattice list-targets \
--target-group-identifier "$(terraform output -raw target_group_id)" \
--query 'items[].[id,status,reasonCode]' --output text
# i-0a1b… UNHEALTHY HealthCheckFailedUNHEALTHY is the broken state. If it still says INITIAL, wait a minute.
SVC=$(terraform output -raw service_dns_name)
APP_IP=$(terraform output -raw app_private_ip)
terraform output -raw start_session_commandConfirm the direct path works
On the client:
curl -s -m 5 "http://$APP_IP/"
# lab-27-application-okThe security group rule is doing its job for this packet. The client is in sg-client, the rule
admits sg-client, the request lands.
Confirm the Lattice path does not
dig +short "$SVC"
# 169.254.171.32
# 169.254.171.33
curl -s -m 5 -o /dev/null -w '%{http_code}\n' "http://$SVC/"; echo "exit $?"Two things to read. The service name resolves to 169.254.171.x addresses, which are link-local:
the client is not talking to anything in 10.27.0.0/16. And the request does not complete. Depending
on how the service answers a request with no healthy target, you will see a non-200 status from
Lattice itself or a timeout; either way, nothing reached the application.
Watch what arrives at the application
Open a second session, on the application:
terraform output -raw app_session_commandThe Python server logs every request with its source address. Follow the log while the health checks run:
journalctl -u lab-http -fNothing. The health checker is trying every ten seconds and none of its connections reach the
process, because sg-app is refusing them before the kernel sees a SYN. The log will only show the
direct request you sent earlier, from 10.27.1.x, the client's address.
Now look at the one rule:
aws ec2 describe-security-group-rules \
--filters "Name=group-id,Values=$(terraform output -raw app_security_group_id)" \
--query 'SecurityGroupRules[?!IsEgress].[FromPort,ToPort,ReferencedGroupInfo.GroupId,CidrIpv4,PrefixListId]' \
--output text
# 80 80 sg-0c1… None NoneSource: the client group. Only the client group.
This debrief is part of Labs Pro
The root-cause analysis, packet-flow walkthrough, and Terraform remediation diff for this lab are available to Labs Pro members. One payment of $49, no subscription, and it covers every Pro lab now and later.
The brief, the reproduction steps, and the Terraform stay free — you can still solve this one yourself.
Already bought it? Sign in and it unlocks.