Lab 04: The Endpoint That Answered To Nobody

What you are working with#

Two VPCs. The provider hosts a service behind an internal Network Load Balancer, exposed as a VPC endpoint service. The consumer reaches it through an interface endpoint. There is no peering, no Transit Gateway, and no internet path between them — PrivateLink is the only connection.

Scope and constraints#

• In scope: the interface endpoint and what governs traffic reaching it.
• Out of scope: the endpoint service, the NLB, its target group health, the backend service, DNS, routing, and network ACLs. All are correct — the ticket's checks were accurate.
• The consumer's application security group is not the problem. It permits all egress and widening it further will not help. Read that sentence carefully; it is true and it is also a hint.
• You get a shell on both hosts. The backend is reachable directly from within the provider VPC if you want to confirm the service works.

Deploy the broken state#

terraform init
terraform apply

# Shell on the consumer application host
terraform output -raw consumer_session_command

# The URL the application uses
terraform output -raw endpoint_url

Full source: main.tf. It builds both VPCs, the internal NLB with a health-checked target group, the endpoint service with the consumer registered as an allowed principal, the interface endpoint, and one instance per side.

Two implementation notes so they do not read as bugs. private_dns_enabled is false because private DNS on a custom endpoint service requires domain ownership verification — consumers use the generated endpoint DNS name, which is the normal pattern for third-party services. And the target group sets preserve_client_ip = "false" so the backend sees a source address inside the provider VPC, keeping backend security group rules from becoming a second, unintended failure.

Get the target:

terraform output -raw endpoint_url
# http://vpce-0abc123-def456.vpce-svc-0789xyz.us-east-1.vpce.amazonaws.com:8080/

Open a shell on the consumer application host:

aws ssm start-session --region us-east-1 --target <app instance id>

Confirm the failure#

URL=$(terraform output -raw endpoint_url)   # or paste it in
curl -sS -m 10 "$URL"
# curl: (28) Connection timed out after 10002 milliseconds

A timeout. By now that should immediately tell you the packet is being discarded somewhere rather than actively refused.

Confirm DNS is not the problem#

ENDPOINT_HOST=vpce-0abc123-def456.vpce-svc-0789xyz.us-east-1.vpce.amazonaws.com
 
dig +short "$ENDPOINT_HOST"
# 10.20.1.87

That address is inside the consumer subnet — it is the endpoint's own network interface. DNS is doing exactly the right thing, and the destination is one hop away on the local subnet.

Connect straight to the address to take DNS out of the picture entirely:

nc -vz -w 8 10.20.1.87 8080
# Ncat: Connection timed out.

Confirm the service itself works#

Worth eliminating so you are not chasing a broken backend. Open a shell on the provider host and hit the service locally, then through the NLB:

# On the provider backend host
curl -sS -m 5 http://localhost:8080/
# vn-lab-04 provider backend ok

And confirm the target group agrees:

aws elbv2 describe-target-health \
  --target-group-arn "$(terraform output -raw target_group_arn)" \
  --query 'TargetHealthDescriptions[].TargetHealth.State' --output text
# healthy

The service is up and the load balancer knows it.

Look at the endpoint itself#

The team verified the endpoint's state. Look at its configuration:

aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids "$(terraform output -raw vpc_endpoint_id)" \
  --query 'VpcEndpoints[0].{State:State,Type:VpcEndpointType,Groups:Groups,NetworkInterfaces:NetworkInterfaceIds}' \
  --output json

Read the Groups field. Then compare it against the application's own security group:

echo "app sg:             $(terraform output -raw app_security_group_id)"
echo "consumer default:   $(terraform output -raw consumer_default_security_group_id)"

If those differ, work out what the group attached to the endpoint actually permits:

aws ec2 describe-security-groups \
  --group-ids "$(terraform output -raw consumer_default_security_group_id)" \
  --query 'SecurityGroups[0].IpPermissions' --output json

Root cause#

An interface endpoint is not an abstract routing construct. It creates elastic network interfaces in your subnets, and those ENIs have their own security groups — independent of the security group on whatever is calling them.

The endpoint was created without specifying one, so AWS attached the VPC default security group. The default group's only inbound rule permits traffic from members of the default group itself. The application host is in sg-app, not the default group, so its traffic matches nothing and is dropped at the endpoint ENI:

aws ec2 describe-security-groups --group-ids sg-0default… \
  --query 'SecurityGroups[0].IpPermissions' --output json
# [
#   {
#     "IpProtocol": "-1",
#     "IpRanges": [],
#     "UserIdGroupPairs": [ { "GroupId": "sg-0default…" } ]   ← itself, and only itself
#   }
# ]

That is the entire defect. The application's egress rules were never relevant, because the traffic was not blocked on the way out — it was blocked on arrival, one hop later, at a resource nobody had listed.

Why "the endpoint is available" was misleading#

State: available describes the endpoint's provisioning lifecycle — the ENIs exist, the service accepted the connection, DNS is published. It says nothing about whether traffic is permitted to reach those ENIs. This is the same category of mistake as Lab 01, where a NAT Gateway reported available while having no path to the internet: a healthy control-plane state is not a statement about the data plane.

Worth building the habit of asking, for any managed endpoint: what state is being reported, and is it about provisioning or about reachability? It is almost always the former.

The general principle#

This generalizes well beyond PrivateLink, which is what makes it worth the time:

| Managed feature | What it puts in your VPC | Consequence | | --- | --- | --- | | Interface VPC endpoint | ENIs in your subnets | Own security group governs who may reach it | | Route 53 Resolver endpoint | ENIs in your subnets | Own security group governs DNS reaching it | | RDS Proxy | ENIs in your subnets | Own security group, distinct from the database's | | Lambda in a VPC | ENIs in your subnets | Own security group governs its egress | | NAT Gateway | An address in a subnet | Forwards per that subnet's route table (Lab 01) |

The pattern: AWS managed features that need to live inside your network do so by placing real interfaces in it, and those interfaces are subject to the same rules as anything else you put there. When something managed is unreachable, find out what it actually materialized as before inspecting anything further away.

Why the default security group is the trap#

Two properties combine badly. It is attached silently when you specify nothing, and its inbound rule is self-referential — it permits members of itself, which reads as "permissive" at a glance and is in practice restrictive for anything you actually deploy. So the failure mode is: omit an optional argument, get a group that looks permissive, and find nothing can reach it.

The fix#

Give the endpoint a security group that permits the service port from the consumers that need it.

terraform apply
# aws_security_group.endpoint will be created
# aws_vpc_endpoint.service will be updated in-place
# Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

The endpoint is updated in place — no new ENIs, no DNS change, no downtime beyond the security group swap taking effect.

Verify#

Same host, same URL:

curl -sS -m 10 "$URL"
# vn-lab-04 provider backend ok

And confirm the endpoint now carries the intended group rather than the default:

aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids "$(terraform output -raw vpc_endpoint_id)" \
  --query 'VpcEndpoints[0].Groups' --output table
# vn-lab-04-endpoint

Senior debrief#

The transferable lesson: when a managed AWS feature has to operate inside your VPC, it does so by creating real network interfaces there — and those interfaces obey the same security group rules as anything else. Before debugging a managed endpoint, find out what it materialized as. describe the thing and read what it actually created.

Why the checklist failed. The team enumerated every resource they had authored, and every one was correct. The blocking resource was one they had not authored — a security group AWS selected on their behalf because they left an optional argument unset. There is no diff to review and no line of Terraform to inspect. This is the failure mode of infrastructure-as-code specifically: the gap between what you wrote and what exists is invisible if you only read what you wrote.

On available. Two of these four labs have now turned on the same misreading. A NAT Gateway reporting available with no route to an internet gateway, and an interface endpoint reporting available that nothing may reach. Control-plane state describes provisioning. It is never a claim about reachability. Treat "the console says it is healthy" as the beginning of an investigation.

How this presents in an interview. "PrivateLink endpoint is available, DNS resolves, connection times out" is a strong senior screening question because the naive answer — check the client's security group and the route table — is both reasonable and wrong. The answer that lands notes that the endpoint has its own ENIs with their own security group, and that the default group is attached when none is specified. Extending it to Route 53 Resolver endpoints and RDS Proxy shows you hold the principle rather than the anecdote.

Guardrails worth adding:

• Make security_group_ids non-optional in your own module wrapper. If every interface endpoint in your estate is created through an internal module, that module can require the argument. This entire class of failure disappears at the module boundary.
• A policy check rejecting any aws_vpc_endpoint of type Interface without security_group_ids. Cheap in tflint, Conftest, or Sentinel, and it catches the omission at plan time rather than during an integration cutover.
• Lock down the default security group explicitly. Many organizations manage aws_default_security_group with no rules at all, precisely so that accidentally landing in it fails loudly and immediately rather than looking plausible.
• Test reachability during a migration cutover, not configuration. CHG-2410 replaced a working public egress path with PrivateLink. A single synthetic request through the new path before switching traffic would have caught this before the integration broke.

Clean up#

terraform destroy

The NLB and the endpoint are the hourly line items. Confirm both are gone:

aws ec2 describe-vpc-endpoints \
  --query 'VpcEndpoints[?State!=`deleted`].[VpcEndpointId,State]' --output table
# Should be empty.