Skip to content
VenerableNetworks
SpecialtyProRoute 53 Resolver

Lab 12: The Name That Resolved To Last Year

The private hosted zone is associated with the VPC and its record is correct. The resolver is healthy. Every connection succeeds — to the host that was decommissioned during the migration six months ago.

Debugging time
~35 min
Reading time
11 min
Reported by
Payments Platform
Tier
Specialty
INC-1566SEV-1InvestigatingOpened 2026-09-21 03:52 UTC

Payments reconciliation is writing to the decommissioned legacy host

Reported by Payments Platform

Reconciliation has been writing to the old payments host. We found it because the legacy box filled its disk, which is the only reason anyone noticed — it has been answering requests for months.

payments.corp.internal is supposed to point at the current host. The record in the private hosted zone is correct; I have checked it three times in the Route 53 console. The zone is associated with the VPC.

The application resolves the name and connects successfully. It just connects to the wrong machine. There is no error anywhere. Every health check is green on both hosts.

Nobody has touched DNS in this account since the migration.

What you are working with

One VPC, two Availability Zones, and a hybrid DNS setup that predates the migration.

| Resource | Configuration | | --- | --- | | VPC | 10.100.0.0/16, DNS support and hostnames enabled | | Private hosted zone | corp.internal, associated with the VPC | | Record | payments.corp.internal → current host, TTL 60 | | Resolver outbound endpoint | Two ENIs, OPERATIONAL | | Resolver rule | FORWARD for corp.internal → corporate resolver | | Rule association | Associated with this VPC | | Corporate resolver | Answers payments.corp.internal with the legacy host |

Two things claim the same domain. That is the entire lab, and it is visible in the table above before you run a single command.

  1. EC2

    Application

    10.100.1.x — queries payments.corp.internal

  2. RTB

    Route 53 Resolver, 10.100.0.2

    evaluates rules before hosted zones

  3. GW

    Outbound endpoint ENI

    forwards the query to the corporate resolver

  4. VPCE

    Corporate resolver

    answers with the legacy host address

  5. DEST

    Legacy payments host

    accepts the connection and serves the request

Nothing drops. Every hop succeeds. That is what makes this worse than an outage — the system is working correctly and doing the wrong thing.

Scope and constraints

  • In scope: which of the two DNS sources answers for this name, and why.
  • Out of scope: the private hosted zone's record, its VPC association, the resolver endpoint's health, and both hosts. All are correct and working.
  • enable_dns_support and enable_dns_hostnames are on.
  • The record in the private hosted zone really is right. The reporter checked it three times and was correct every time.
  • The fix is not deleting the Resolver rule. The VPC still needs to resolve data centre names.

Deploy the broken state

cd lab-12-resolver-rule-precedence
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw app_instance_id)"

The Resolver endpoint takes several minutes to become operational, and dnsmasq needs a moment on the simulated corporate resolver. Give the apply five minutes before testing.