Skip to content
VenerableNetworks
SpecialtyProRoute 53 ResolverAWS VPC

Lab 05: The Hostname That Existed In Only One VPC

A peered VPC cannot resolve an internal service name. The peering is active, routes exist both ways, security groups permit the port, and connecting by IP address works perfectly. The record is present and correct in the private hosted zone.

Debugging time
~30 min
Reading time
11 min
Reported by
Data Platform
Tier
Specialty
INC-1341SEV-3InvestigatingOpened 2026-09-16 09:12 UTC

Reporting service cannot resolve db.svc.internal after migration to its own VPC

Reported by Data Platform

The reporting service moved out of the shared VPC into its own account-level VPC last night. Everything about the move looked clean. The VPC peering connection is active, routes were added in both directions, and the database security group already permits the new CIDR on 5432.

The service will not start. It fails at connection setup with a name resolution error, not a timeout — so it never gets as far as the network. A developer confirmed she can reach the database by IP address from the new VPC without any problem.

Nobody has changed the hosted zone. The record is there; she checked it in the console.

What you are working with

Two VPCs, peered, in a single Availability Zone.

Starting state — routing is symmetric and the peering is active
VPC app 10.30.0.0/16 AZ a contains app 10.30.1.0/24, default route 10.31.0.0/16 → pcx, holding Reporting host. VPC data 10.31.0.0/16 AZ a contains data 10.31.1.0/24, default route 10.30.0.0/16 → pcx, holding Database host.VPC peeringactive bothwaysVPC app 10.30.0.0/16AZ aapppublic10.30.1.0/2410.31.0.0/16 → pcxReporting hostsg: all egressVPC data 10.31.0.0/16AZ adatapublic10.31.1.0/2410.30.0.0/16 → pcxDatabase hostlistens on 5432

Each route table carries the other VPC's CIDR at the peering connection, so packets cross in both directions. The data security group admits 5432 from 10.30.0.0/16, which is the app VPC exactly. Every object on this diagram is correct — and the thing the application actually needs is not drawn here at all, because it is not part of the packet path.

  1. EC2

    Reporting host (app VPC)

    10.30.1.x — resolving db.svc.internal

  2. RTB

    Route 53 Resolver, app VPC

    10.30.0.2 — answers for zones associated with THIS VPC

    Dropped — svc.internal is not associated with the app VPC, so the Resolver has no record to return and falls through to public DNS, which has no .internal

  3. GW

    VPC peering

    active, routed both ways — never reached

  4. DEST

    Database host (data VPC)

    10.31.1.x:5432 — listening and reachable by IP

The failure happens at the first hop, before any packet is addressed to the database. Everything downstream is healthy, which is why every network-level check passes.

ResourceConfiguration
App VPC10.30.0.0/16, DNS support and hostnames both enabled
Data VPC10.31.0.0/16, DNS support and hostnames both enabled
Peeringactive, 10.31.0.0/16 routed from app, 10.30.0.0/16 routed from data
Private hosted zonesvc.internal, containing db.svc.internal → 10.31.1.x
sg-dataInbound 5432 from 10.30.0.0/16
sg-appAll outbound

Both instances sit in public subnets with an internet gateway, purely so dig and nc install at launch. That has no bearing on the failure.

Scope and constraints

  • In scope: DNS resolution from the application VPC.
  • Out of scope: the database itself, the peering connection, security groups, and route tables. All four are correctly configured and you should be able to prove it quickly.
  • enable_dns_support and enable_dns_hostnames are on for both VPCs. This is not that bug.
  • The record exists and its value is correct. This is not a missing-record bug.
  • Reaching for a Route 53 Resolver endpoint is the expensive wrong answer here. Part of the exercise is being able to say why.

Deploy the broken state

cd lab-05-private-zone-association
terraform init
terraform apply
 
# Shell on the reporting host
aws ssm start-session --target "$(terraform output -raw app_instance_id)"
 
# The values you will need
terraform output service_fqdn
terraform output data_private_ip
terraform output app_vpc_id
terraform output hosted_zone_id

Allow a minute after apply for user_data to install the tooling and start the listener.