Lab 10: The Endpoint That Resolved To The Internet

What you are working with#

One VPC, one Availability Zone, one genuinely private subnet. There is no internet gateway anywhere in this configuration.

| Resource | Configuration | | --- | --- | | VPC | 10.80.0.0/16, DNS support and hostnames both enabled | | Subnet | 10.80.1.0/24, route table has the local route only | | vpce-ssm, vpce-ssmmessages, vpce-ec2messages | Interface, available, private DNS enabled | | vpce-secretsmanager | Interface, available, private DNS disabled | | sg-endpoints | Inbound 443 from 10.80.0.0/16 — shared by all four endpoints | | Instance role | AmazonSSMManagedInstanceCore plus GetSecretValue on the one secret |

The security group is shared across all four endpoints. That is the single most useful fact in the table, because it means any explanation involving the security group has to explain why three endpoints work and one does not.

Scope and constraints#

• In scope: why the SDK's traffic never reaches the endpoint.
• Out of scope: the endpoint's security group, its subnet placement, its state, and the IAM policy. All four are correct, and three of them are provably correct because Session Manager depends on the same ones.
• enable_dns_support and enable_dns_hostnames are both on. This is not that bug.
• The endpoint is available, not pending or rejected.
• This is not Lab 04. There the endpoint had the wrong security group. Here the security group is right and is shared with endpoints that work.

Deploy the broken state#

cd lab-10-endpoint-private-dns
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw host_instance_id)"

Interface endpoints take a couple of minutes to become available, and Session Manager will not connect until the three SSM endpoints are ready.

SECRET=$(terraform output -raw secret_name)
HOST=$(terraform output -raw service_hostname)   # secretsmanager.us-east-1.amazonaws.com

Confirm the failure#

From the session on the worker:

time aws secretsmanager get-secret-value --secret-id "$SECRET" --region us-east-1
# Connect timeout on endpoint URL: "https://secretsmanager.us-east-1.amazonaws.com/"
#
# real	1m5.284s

A connect timeout, not an access denied and not a TLS failure. The client never established a TCP connection, which puts this below the application layer entirely.

Prove the subnet works#

You are already using the proof, but make it explicit:

aws ssm describe-instance-information --region us-east-1 \
  --query 'InstanceInformationList[0].InstanceId' --output text
# i-0bb22cc33dd44ee55

That call went out from this host, over an interface endpoint, in this subnet, through sg-endpoints, and returned. So the subnet reaches interface endpoints, the endpoint security group permits 443, and the resolver works.

Resolve the two hostnames and compare#

This is the whole lab in two commands.

getent hosts ssm.us-east-1.amazonaws.com
# 10.80.1.171     ssm.us-east-1.amazonaws.com
 
getent hosts secretsmanager.us-east-1.amazonaws.com
# 3.234.16.98     secretsmanager.us-east-1.amazonaws.com
# 44.208.11.7     secretsmanager.us-east-1.amazonaws.com

The Systems Manager hostname resolves to an address inside the subnet. The Secrets Manager hostname resolves to public addresses on the internet.

The worker is not failing to reach its endpoint. It is faithfully reaching for the public API, from a subnet that has no route to anything public.

Confirm there is no route for that address#

ip route get 3.234.16.98
# RTNETLINK answers: Network is unreachable

Or from the AWS side, note what the route table contains:

aws ec2 describe-route-tables --region us-east-1 \
  --filters "Name=association.subnet-id,Values=$(aws ec2 describe-instances \
    --instance-ids "$(curl -s -H "X-aws-ec2-metadata-token: $(curl -sX PUT \
      http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')" \
      http://169.254.169.254/latest/meta-data/instance-id)" \
    --query 'Reservations[0].Instances[0].SubnetId' --output text)" \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,GatewayId,State]' --output table
# --------------------------------------------
# |  10.80.0.0/16  |  local  |  active       |
# --------------------------------------------

One route. Anything not in 10.80.0.0/16 has nowhere to go.

Look at the endpoint's DNS setting#

aws ec2 describe-vpc-endpoints --region us-east-1 \
  --vpc-endpoint-ids "$(terraform output -raw secretsmanager_endpoint_id)" \
  --query 'VpcEndpoints[0].{State:State,PrivateDns:PrivateDnsEnabled,Subnets:SubnetIds,SGs:Groups[].GroupId}'
# {
#     "State": "available",
#     "PrivateDns": false,
#     "Subnets": ["subnet-0aa11bb22"],
#     "SGs": ["sg-0cc33dd44"]
# }

available, correct subnet, correct security group, and PrivateDnsEnabled: false.

Prove the endpoint itself is healthy#

The endpoint always has its own DNS names, whether or not it claims the service hostname. Use one directly:

terraform output secretsmanager_endpoint_dns
# [
#   { "dns_name" = "vpce-0a1b2c3d-4e5f6g7h.secretsmanager.us-east-1.vpce.amazonaws.com", ... },
#   { "dns_name" = "vpce-0a1b2c3d-4e5f6g7h-us-east-1a.secretsmanager.us-east-1.vpce.amazonaws.com", ... }
# ]

aws secretsmanager get-secret-value --secret-id "$SECRET" --region us-east-1 \
  --endpoint-url https://vpce-0a1b2c3d-4e5f6g7h.secretsmanager.us-east-1.vpce.amazonaws.com \
  --query SecretString --output text
# {"username":"lab","password":"lab-10-secret-ok"}

It works. Same endpoint, same security group, same subnet, same IAM role — reached by a name that resolves to it. The endpoint was never the problem, and this also confirms the IAM policy is fine.