Lab 12: The Name That Resolved To Last Year

What you are working with#

One VPC, two Availability Zones, and a hybrid DNS setup that predates the migration.

Two things claim the same domain. That is the entire lab, and it is visible in the table above before you run a single command.

Scope and constraints#

• In scope: which of the two DNS sources answers for this name, and why.
• Out of scope: the private hosted zone's record, its VPC association, the resolver endpoint's health, and both hosts. All are correct and working.
• enable_dns_support and enable_dns_hostnames are on.
• The record in the private hosted zone really is right. The reporter checked it three times and was correct every time.
• The fix is not deleting the Resolver rule. The VPC still needs to resolve data centre names.

Deploy the broken state#

cd lab-12-resolver-rule-precedence
terraform init
terraform apply
 
aws ssm start-session --target "$(terraform output -raw app_instance_id)"

The Resolver endpoint takes several minutes to become operational, and dnsmasq needs a moment on the simulated corporate resolver. Give the apply five minutes before testing.

FQDN=$(terraform output -raw service_fqdn)          # payments.corp.internal
CURRENT=$(terraform output -raw payments_current_ip)
LEGACY=$(terraform output -raw onprem_ip)

Confirm the wrong host is answering#

From the application host:

nc -w 3 "$FQDN" 5000
# lab-12-payments-LEGACY-do-not-use

The connection succeeds. The banner says it reached the legacy host. Compare the two directly:

nc -w 3 "$CURRENT" 5000
# lab-12-payments-CURRENT-ok
 
nc -w 3 "$LEGACY" 5000
# lab-12-payments-LEGACY-do-not-use

Both hosts are up and both answer. The question is purely which address the name produces.

Ask what the name resolves to#

dig +short "$FQDN"
# 10.100.2.87

echo "current: $CURRENT   legacy: $LEGACY"
# current: 10.100.1.44   legacy: 10.100.2.87

The name resolves to the legacy address. Nothing about the connection is wrong — the application is faithfully connecting to the address it was given.

Confirm the private hosted zone is correct#

This is the step the reporter did, and they were right.

aws route53 list-resource-record-sets --region us-east-1 \
  --hosted-zone-id "$(terraform output -raw hosted_zone_id)" \
  --query "ResourceRecordSets[?Name=='payments.corp.internal.'].[Name,Type,ResourceRecords[0].Value]" \
  --output text
# payments.corp.internal.	A	10.100.1.44

The zone says 10.100.1.44 — the current host. And the association is real:

aws route53 get-hosted-zone --region us-east-1 \
  --id "$(terraform output -raw hosted_zone_id)" \
  --query 'VPCs[].VPCId' --output text
# vpc-0aa11bb22cc33dd44

So the zone holds the right answer, is associated with the right VPC, and the resolver in that VPC is returning a different answer. Both facts are true at once, which is the point where most people start re-checking things they have already checked.

Ask the resolver where the answer came from#

The header flags are the tell.

dig +noall +comments +answer "$FQDN"
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51204
# ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
#
# payments.corp.internal.	0	IN	A	10.100.2.87

Note the TTL: 0. The record in the private hosted zone has a TTL of 60, so this answer did not come from the hosted zone — something else served it.

Be careful about how far you generalise that, because it is the one piece of evidence here that is an artifact of the lab rather than a property of AWS. The Resolver relays whatever the upstream sent, and the upstream here is the dnsmasq standing in for the corporate resolver, which answers its own address= entries with a TTL of 0 by default. A real corporate resolver — BIND, Windows DNS, Infoblox — would return its own record's TTL, which could easily be 60 as well. A mismatched TTL is a hint worth taking; a matching TTL proves nothing.

The evidence that does generalise is the configuration, which is what the next step reads.

Find the rule#

aws route53resolver list-resolver-rules --region us-east-1 \
  --query 'ResolverRules[?RuleType==`FORWARD`].[Id,DomainName,RuleType,Status]' \
  --output table
# ----------------------------------------------------------------------
# |  rslvr-rr-0a1b2c3d  |  corp.internal.  |  FORWARD  |  COMPLETE    |
# ----------------------------------------------------------------------

A forwarding rule for corp.internal — the same domain as the private hosted zone. Confirm it applies to this VPC and see where it points:

aws route53resolver list-resolver-rule-associations --region us-east-1 \
  --query 'ResolverRuleAssociations[].[ResolverRuleId,VPCId,Status]' --output text
# rslvr-rr-0a1b2c3d	vpc-0aa11bb22cc33dd44	COMPLETE
 
aws route53resolver get-resolver-rule --region us-east-1 \
  --resolver-rule-id rslvr-rr-0a1b2c3d \
  --query 'ResolverRule.TargetIps[].Ip' --output text
# 10.100.2.87

The rule forwards to the corporate resolver, which is also the legacy host in this lab. Ask that resolver directly and you get the stale record:

dig +short "$FQDN" @"$LEGACY"
# 10.100.2.87