Skip to content
VenerableNetworks
ProfessionalProPrivateLinkRoute 53 Resolver

Lab 10: The Endpoint That Resolved To The Internet

A private workload cannot read its secret at startup. The Secrets Manager interface endpoint is available, in the right subnet, and its security group permits 443 from the VPC. Session Manager works through endpoints in that same subnet, with that same security group.

Debugging time
~25 min
Reading time
11 min
Reported by
Payments Platform
Tier
Professional
INC-1512SEV-2InvestigatingOpened 2026-09-19 06:31 UTC

Payments worker times out fetching its database credential on boot

Reported by Payments Platform

The worker runs in a private subnet with no internet route, by design. It reads its database password from Secrets Manager at startup and reaches the API through an interface endpoint.

The endpoint is available. It has an ENI in the worker's subnet. Its security group allows 443 from the VPC CIDR. The instance role has secretsmanager:GetSecretValue on exactly that secret.

Every GetSecretValue hangs and eventually times out. No permission error, no TLS error, no throttling. Just nothing.

We know the subnet's networking is functional because Session Manager works — we are debugging this over a session on the affected host right now, and Session Manager goes through interface endpoints in the same subnet using the same security group.

What you are working with

One VPC, one Availability Zone, one genuinely private subnet. There is no internet gateway anywhere in this configuration.

Starting state — four endpoints in one subnet, sharing one security group
VPC 10.80.0.0/16 AZ a contains private 10.80.1.0/24, default route local only, holding Host, vpce-ssm, vpce-ssmmessages, vpce-ec2messages, vpce-secretsmanager.VPC 10.80.0.0/16AZ aprivateisolated10.80.1.0/24local onlyHostno egress, no toolingvpce-ssmprivate DNS onvpce-ssmmessagesprivate DNS onvpce-ec2messagesprivate DNS onvpce-secretsmanagerprivate DNS off

There is no internet gateway, no NAT Gateway and no egress-only gateway in this configuration, and the route table holds nothing but the implicit local route — so anything that resolves to a public address is unreachable by construction. All four endpoints sit in the same subnet and share one security group admitting 443 from the VPC CIDR, which rules out both placement and filtering as explanations. The four differ in exactly one attribute.

ResourceConfiguration
VPC10.80.0.0/16, DNS support and hostnames both enabled
Subnet10.80.1.0/24, route table has the local route only
vpce-ssm, vpce-ssmmessages, vpce-ec2messagesInterface, available, private DNS enabled
vpce-secretsmanagerInterface, available, private DNS disabled
sg-endpointsInbound 443 from 10.80.0.0/16 — shared by all four endpoints
Instance roleAmazonSSMManagedInstanceCore plus GetSecretValue on the one secret

The security group is shared across all four endpoints. That is the single most useful fact in the table, because it means any explanation involving the security group has to explain why three endpoints work and one does not.

  1. EC2

    Worker host

    10.80.1.x — resolving secretsmanager.us-east-1.amazonaws.com

  2. RTB

    Route 53 Resolver, 10.80.0.2

    answers with the public regional addresses for the service

  3. SUBNET

    Route lookup for a public address

    route table: 10.80.0.0/16 → local, and nothing else

    Dropped — The destination is a public IP. No route matches, so the packet is discarded inside the VPC.

  4. VPCE

    vpce-secretsmanager ENI

    10.80.1.x:443 — available, healthy, never addressed

The endpoint is not broken and was never contacted. The client resolved the service to an address the endpoint does not answer on.

Scope and constraints

  • In scope: why the SDK's traffic never reaches the endpoint.
  • Out of scope: the endpoint's security group, its subnet placement, its state, and the IAM policy. All four are correct, and three of them are provably correct because Session Manager depends on the same ones.
  • enable_dns_support and enable_dns_hostnames are both on. This is not that bug.
  • The endpoint is available, not pending or rejected.
  • This is not Lab 04. There the endpoint had the wrong security group. Here the security group is right and is shared with endpoints that work.

Deploy the broken state

cd lab-10-endpoint-private-dns
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw host_instance_id)"

Interface endpoints take a couple of minutes to become available, and Session Manager will not connect until the three SSM endpoints are ready.