Skip to content
VenerableNetworks
SpecialtyFree accountPrivateLinkNACL / Security GroupsAWS VPC

Lab 04: The Endpoint That Answered To Nobody

A PrivateLink interface endpoint reports available, DNS resolves to it, the load balancer behind it has a healthy target, and the application security group allows all egress. Every connection still times out, and nothing in the consumer configuration looks wrong.

Debugging time
~30 min
Reading time
14 min
Reported by
Integrations
Tier
Specialty
INC-1207SEV-3InvestigatingOpened Today 09:31 UTC

Application cannot reach the vendor API through its new PrivateLink endpoint

Reported by Integrations

Environment
staging
Region
us-east-1
Consumer VPC
10.20.0.0/16
Endpoint type
Interface (PrivateLink)
Change ref
CHG-2410 — replace public egress with PrivateLink

We moved the vendor API integration off public internet egress and onto PrivateLink, so the traffic never leaves AWS. The endpoint came up clean. The application cannot connect.

What we have confirmed:

  • The VPC endpoint state is available, not pending or pendingAcceptance.
  • The endpoint DNS name resolves from the application host, to an address inside our own subnet.
  • The application's security group allows all egress. We widened it to be certain.
  • There are no custom network ACLs on the subnet — it is the VPC default, allow all.
  • The route table is untouched. The endpoint address is in our own VPC CIDR, so it is a local route.
  • On the provider side the load balancer target is healthy.

Every layer we know how to check reports healthy, and curl hangs until it times out. We are starting to think PrivateLink is broken, which we accept is unlikely.

What you are working with

Two VPCs. The provider hosts a service behind an internal Network Load Balancer, exposed as a VPC endpoint service. The consumer reaches it through an interface endpoint. There is no peering, no Transit Gateway, and no internet path between them — PrivateLink is the only connection.

Starting state — PrivateLink is the only path between the two VPCs
VPC consumer 10.20.0.0/16 AZ a contains consumer 10.20.1.0/24, holding App host, Endpoint ENI. VPC provider 10.10.0.0/16 AZ a contains provider 10.10.1.0/24, holding Internal NLB, Backend host.endpointserviceprincipalallowedVPC consumer 10.20.0.0/16AZ aconsumerpublic10.20.1.0/24App hostsg: allegressEndpointENIavailableVPC provider 10.10.0.0/16AZ aproviderpublic10.10.1.0/24InternalNLBTCP 8080Backendhosthealthytarget

Both subnets are public only so Session Manager can reach the hosts; the failing request never uses that path. The backend security group permits 8080 from all of 10.0.0.0/8 and the application's own security group permits all egress, so neither of the two security groups written in the Terraform is restricting this flow. Note what the endpoint's ENI is drawn as: an interface in the consumer's subnet, which means it has security groups of its own.

10.20.1.30:41902 → endpoint ENI:8080 (TCP SYN)
  1. EC2

    Application host

    10.20.1.30 in the consumer VPC

  2. FILTER

    Application security group — egress

    sg-app — allow all outbound

  3. RTB

    Consumer VPC route table

    10.20.0.0/16 → local — the endpoint ENI is in this subnet

  4. VPCE

    Interface endpoint ENI

    vpce-… — state: available

  5. GW

    Endpoint service → NLB

    healthy target in the provider VPC

  6. DEST

    Service backend

    10.10.1.20:8080 — listening

Everything the team checked is on this diagram and all of it passes. The hop they did not think to inspect is the endpoint itself.

Scope and constraints

  • In scope: the interface endpoint and what governs traffic reaching it.
  • Out of scope: the endpoint service, the NLB, its target group health, the backend service, DNS, routing, and network ACLs. All are correct — the ticket's checks were accurate.
  • The consumer's application security group is not the problem. It permits all egress and widening it further will not help. Read that sentence carefully; it is true and it is also a hint.
  • You get a shell on both hosts. The backend is reachable directly from within the provider VPC if you want to confirm the service works.

Deploy the broken state

main.tf8 lines
Download

Download the file below into an empty directory and apply it. Expect five to seven minutes — the NLB and the endpoint service both take a while to become available.

terraform init
terraform apply

# Shell on the consumer application host
terraform output -raw consumer_session_command

# The URL the application uses
terraform output -raw endpoint_url

Full source: main.tf. It builds both VPCs, the internal NLB with a health-checked target group, the endpoint service with the consumer registered as an allowed principal, the interface endpoint, and one instance per side.

Two implementation notes so they do not read as bugs. private_dns_enabled is false because private DNS on a custom endpoint service requires domain ownership verification — consumers use the generated endpoint DNS name, which is the normal pattern for third-party services.

The target group also sets preserve_client_ip = "false", and it is worth being precise about why, because the obvious reading is wrong. It is not what stops the backend seeing a consumer address. AWS documents that client IP preservation has no effect on inbound traffic from PrivateLink: the source address of PrivateLink traffic is always the private address of the Network Load Balancer, whichever way the attribute is set. A consumer's private IP never crosses the PrivateLink boundary, so the backend could not see one to begin with.

Disabling it is therefore a no-op for this lab's data path. What it does buy you in a real deployment is documented on the same page: with preservation off, an NLB supports roughly 55,000 simultaneous connections per unique target, and exceeding that produces port allocation errors. It is set here for symmetry with how you would build the provider side for real, not because the lab depends on it.