Skip to content
VenerableNetworks
SpecialtyProRoute 53 Resolver

Lab 12: The Name That Resolved To Last Year

The private hosted zone is associated with the VPC and its record is correct. The resolver is healthy. Every connection succeeds — to the host that was decommissioned during the migration six months ago.

Debugging time
~35 min
Reading time
13 min
Reported by
Payments Platform
Tier
Specialty
INC-1566SEV-1InvestigatingOpened 2026-09-21 03:52 UTC

Payments reconciliation is writing to the decommissioned legacy host

Reported by Payments Platform

Reconciliation has been writing to the old payments host. We found it because the legacy box filled its disk, which is the only reason anyone noticed — it has been answering requests for months.

payments.corp.internal is supposed to point at the current host. The record in the private hosted zone is correct; I have checked it three times in the Route 53 console. The zone is associated with the VPC.

The application resolves the name and connects successfully. It just connects to the wrong machine. There is no error anywhere. Every health check is green on both hosts.

Nobody has touched DNS in this account since the migration.

What you are working with

One VPC, two Availability Zones, and a hybrid DNS setup that predates the migration.

Starting state — two authorities for corp.internal, both attached to the same VPC
VPC 10.100.0.0/16 Attached: Internet gateway, Route 53 Resolver 10.100.0.2. AZ a contains subnet-1 10.100.1.0/24, default route 0.0.0.0/0, holding Application host, payments, current, Resolver ENI. AZ b contains subnet-2 10.100.2.0/24, holding Corporate resolver, payments, legacy, Resolver ENI.the internetVPC 10.100.0.0/16AZ asubnet-1public10.100.1.0/24Application hostresolves the namepayments, currentlistens on 5000Resolver ENIoutbound endpointAZ bsubnet-2public10.100.2.0/24Corporate resolverdnsmasq on 53payments, legacylistens on 5000Resolver ENIoutbound endpoint0.0.0.0/0Internet gatewaytooling onlyRoute 53 Resolver10.100.0.2zone and rule both attached

The outbound endpoint needs two addresses in separate zones, which is the only reason there are two subnets. Both answers for payments.corp.internal live inside this VPC: the current host in the first zone, published through the private hosted zone, and the legacy host in the second, published by the dnsmasq process standing in for the corporate resolver. The private hosted zone and the forwarding rule are both attached to this VPC, at the same domain.

ResourceConfiguration
VPC10.100.0.0/16, DNS support and hostnames enabled
Private hosted zonecorp.internal, associated with the VPC
Recordpayments.corp.internal → current host, TTL 60
Resolver outbound endpointTwo ENIs, OPERATIONAL
Resolver ruleFORWARD for corp.internal → corporate resolver
Rule associationAssociated with this VPC
Corporate resolverAnswers payments.corp.internal with the legacy host

Two things claim the same domain. That is the entire lab, and it is visible in the table above before you run a single command.

  1. EC2

    Application

    10.100.1.x — queries payments.corp.internal

  2. RTB

    Route 53 Resolver, 10.100.0.2

    evaluates rules before hosted zones

  3. GW

    Outbound endpoint ENI

    forwards the query to the corporate resolver

  4. VPCE

    Corporate resolver

    answers with the legacy host address

  5. DEST

    Legacy payments host

    accepts the connection and serves the request

Nothing drops. Every hop succeeds. That is what makes this worse than an outage — the system is working correctly and doing the wrong thing.

Scope and constraints

  • In scope: which of the two DNS sources answers for this name, and why.
  • Out of scope: the private hosted zone's record, its VPC association, the resolver endpoint's health, and both hosts. All are correct and working.
  • enable_dns_support and enable_dns_hostnames are on.
  • The record in the private hosted zone really is right. The reporter checked it three times and was correct every time.
  • The fix is not deleting the Resolver rule. The VPC still needs to resolve data centre names.

Deploy the broken state

cd lab-12-resolver-rule-precedence
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw app_instance_id)"

The Resolver endpoint takes several minutes to become operational, and dnsmasq needs a moment on the simulated corporate resolver. Give the apply five minutes before testing.