A Resolver rule outranks your private hosted zone, and TTL 0 is the tell

You can configure two things in one VPC that both claim to answer for corp.internal: a private hosted zone, and a Route 53 Resolver forwarding rule. Nothing warns you. Both are valid, both are associated, and one of them silently never gets consulted.

The evaluation order, per AWS troubleshooting guidance, is most-specific-match first. Where specificity is equal:

1. Resolver rule
2. Private hosted zone
3. Public hosted zone

And stated directly elsewhere: if a Resolver forwarder rule and a private hosted zone conflict, the Resolver rule takes precedence.

The failure this produces#

A migration moves a service into AWS. A private hosted zone is created for corp.internal with the new address, associated with the VPC, and verified in the console. Correct.

What nobody looks at is a forwarding rule for corp.internal created eighteen months earlier so the VPC could resolve data centre names. It is still associated. It is still doing its job.

So every lookup under corp.internal is forwarded to the corporate resolver, which still holds the pre-migration record. The application resolves the name, connects successfully, and talks to the host that was supposed to be decommissioned.

Nothing fails. There is no error, no elevated latency, no failed health check. The system is working correctly and doing the wrong thing, which is worse than an outage because the diagnostic instinct that serves you everywhere else — follow the failing request — has nothing to grab.

Why the obvious check passes#

"Is the DNS record right?" is the correct first question, and the answer is yes. Verifying a record in the Route 53 console tells you what the zone contains. It tells you nothing about whether the zone is being consulted.

Both facts are simultaneously true: the zone holds the right answer, and the resolver returns a different one. That is the point where people start re-checking things they have already checked.

TTL 0#

Here is the cheap signal. A forwarded answer comes back with TTL 0, because the Resolver is relaying a response rather than serving one it is authoritative for.

dig +noall +answer payments.corp.internal
# payments.corp.internal.	0	IN	A	10.100.2.87

If the record in your private hosted zone has a TTL of 60 and the answer arrives with a TTL of 0, the answer did not come from that zone. One field, one dig, and it distinguishes "the zone is wrong" from "the zone is not being read" — which are completely different investigations.

After fixing it, the same check confirms the source changed rather than just the value:

dig +noall +answer payments.corp.internal
# payments.corp.internal.	60	IN	A	10.100.1.44

Address and TTL both moved. Check both.

The fix is specificity, not deletion#

Deleting the forwarding rule is the tempting fix and it is wrong. The VPC genuinely needs data centre names, and removing the rule trades a silent misresolution for a loud outage across a set of services you probably cannot enumerate.

Make one claim more specific so specificity breaks the tie instead of precedence.

Narrow the forwarding rule. Leave corp.internal to the private hosted zone and forward only what actually lives on-premises — dc.corp.internal, for instance. This is the right shape when you control the namespace.

Or add a SYSTEM rule for the exception. A SYSTEM rule means "do not forward this". Being more specific, it beats the broad FORWARD rule:

resource "aws_route53_resolver_rule" "keep_local" {
  name        = "keep-payments-local"
  domain_name = "payments.corp.internal"
  rule_type   = "SYSTEM"
}

This scales badly — one rule per migrated name — but it is exactly right during a migration where names move one at a time.

Reverse DNS collides the other way#

AWS auto-creates reverse lookup rules for your VPC CIDRs. A broad 10.in-addr.arpa forwarding rule aimed at a corporate resolver is less specific than those, so it loses.

That is the mirror image of the forward case and it surprises people in the opposite direction: they configure reverse forwarding, it appears to do nothing, and the reason is that the auto-created rules are more specific. Forwarding reverse zones means disabling the auto-created ones explicitly.

The design fix#

The lasting answer is not a narrower rule; it is deciding that a namespace has exactly one authority. The convention that survives contact with reality is a dedicated subdomain per location — aws.corp.internal for private hosted zones, dc.corp.internal for forwarded names. Then no rule and no zone ever compete, and the precedence order becomes trivia you never need to know.

Instrument it#

Route 53 Resolver query logging records the query, the response, and whether it was forwarded. With it enabled, this incident is one query away: every corp.internal lookup shows as forwarded and the answers do not match the hosted zone. Without it there is no record that DNS did anything at all, which is how six months pass.

And because this class of bug returns a valid answer, the alarm has to assert a value rather than detect an error:

test "$(dig +short payments.corp.internal)" = "$EXPECTED_IP" \
  || echo "FAIL: payments resolves to the wrong host"

"What would tell us if this were quietly wrong?" is a different question from "what would tell us if this broke", and only the second one usually gets built.

Two labs sit on either side of this. The hostname that existed in only one VPC is a zone that was never associated, where reaching for a Resolver endpoint would be an expensive way to paper over a free fix. The name that resolved to last year is a correctly associated zone being outranked by an endpoint that exists for a legitimate reason. Same two components, opposite failures.