A DNS Firewall allowlist blocks the name you allowed, because the CNAME behind it was never on the list

The walled-garden pattern for Route 53 Resolver DNS Firewall is simple to state. One rule allows a list of approved domains. A second rule, lower priority, blocks everything else. Anything in the VPC can resolve the approved names and nothing else.

Then an approved name stops resolving. It is on the list. The list has not changed. The query logs say the firewall blocked it. The thing that changed is on the other side of the internet: the record behind the approved name is now a CNAME.

What the firewall inspects#

From the DNS Firewall overview, under the domain redirection setting:

The domain redirection setting allows you to configure a DNS Firewall rule to inspect all the domains in the DNS redirection chain (default), such as CNAME, DNAME, etc., or just the first domain and trust the rest. If you choose to inspect the entire DNS redirection chain, you must add the subsequent domains to a domain list and set to the action you want the rule to take, either ALLOW, BLOCK, or ALERT.

Two facts in one paragraph. The default is to inspect every name in the chain. And if you keep the default, the names further down the chain need to be on a list too, with an action.

Follow a query under the default. A client asks for app.example.com. The firewall checks the name against the allow rule: match, allowed. The resolver follows the record and finds app.example.com CNAME d111.cloudfront.net. The firewall now checks d111.cloudfront.net. It is not in the allow list. It reaches the block-everything rule, matches, and the query is blocked.

The client asked for a name on the allow list and got the block response. Nothing in the allow list is wrong. The allow list simply does not describe the whole chain, and the firewall was configured, by default, to require that it does.

This is a reasonable default. An allow rule that trusted everything behind the first name would be defeated by any approved domain whose owner later points it somewhere else. The firewall is choosing not to trust an operator you do not control. The cost is that your allow list has to include names you also do not control, and those change.

Why it often appears later#

Nobody builds the allow list with the chain in mind, because when the list is built the chain usually is not there. app.example.com is an A record on day one. On day ninety it moves behind a CDN or a load balancer and becomes a CNAME. The firewall rule has not changed. The query logs show it blocking a name it has allowed for three months.

The same page carries a second detail that catches people who think they have fixed it:

The trust behavior of the domain redirection setting only applies within a single DNS query transaction. If a DNS client on your host separately queries a domain that appears in a DNS redirection chain (for example, querying the redirection target directly), DNS Firewall evaluates it as an independent query with no trust context from the original query. To allow such queries, add the redirection target domains to your domain list.

So switching the rule to trust the chain fixes the lookup of app.example.com. It does not fix a client that, having learned the CNAME target, asks for d111.cloudfront.net directly. That is a new query with no history, and it meets the block rule on its own. Some resolvers and some application stacks do exactly that.

What the client sees#

The block is not an error unless you make it one. From the rule actions page, a BLOCK rule replies with one of:

NODATA – Respond indicating that the query was successful, but no response is available for it. NXDOMAIN – Respond indicating that the query's domain name doesn't exist. OVERRIDE – Provide a custom override in the response.

NODATA looks like a name with no record of that type. NXDOMAIN looks like a name that does not exist. Neither looks like a firewall. An engineer debugging from the client sees a domain they know exists returning NXDOMAIN, concludes the upstream is broken, and opens a ticket with the wrong team. OVERRIDE is the one that announces itself, because it returns a CNAME of your choosing, and the page restricts it: the record type "must be CNAME", and the TTL defaults to zero so the override is not cached.

Separately from the rule's block response, there is what happens when the firewall does not answer at all. From DNS Firewall VPC configuration:

By default, the failure mode is closed, which means that VPC Resolver blocks any queries for which it doesn't receive a reply from DNS Firewall and sends a SERVFAIL DNS response. This approach favors security over availability.

And the scope of that setting:

VPC Resolver enforces a VPC's firewall configuration whenever you have one or more DNS Firewall rule groups associated with the VPC.

Associate a single rule group, even one with a single ALERT rule for testing, and the VPC's resolver is now fail-closed for every query, not just the ones the rule matches. SERVFAIL from the firewall path is a third shape of failure, different from both block responses, and it is the one that affects names no rule mentions.

The query logs are where the shapes become one fact. From the query log format, a firewall-handled query carries firewall_rule_group_id, firewall_rule_action, and firewall_domain_list_id. A log line with those three fields populated and firewall_rule_action: BLOCK is the firewall, regardless of what the client was told.

Evaluation order, in two places#

The CNAME problem is a consequence of the chain. The next class of surprise is a consequence of priority, which exists at two levels. From the overview:

If you associate multiple rule groups with a single VPC, you indicate their processing order through the priority setting in each association. DNS Firewall processes rule groups for a VPC from the lowest numeric priority setting on up.

Each rule in a rule group has a priority setting that's unique within the rule group. DNS Firewall processes the rules in a rule group from the lowest numeric priority setting on up.

And the stop condition:

When DNS Firewall finds a match with a rule's domain list, or anomalies identified by DNS Firewall Advanced rule protections, it terminates the query evaluation and responds to VPC Resolver with the result.

First match wins, across rule groups by association priority and within a rule group by rule priority. An account that associates a shared organisation-wide rule group and a team-specific one has to decide which evaluates first, and the decision is a number on the association, not a property of either group. A team allow rule in a group with association priority 200 never fires for a domain the organisation group at priority 100 already blocked.

What it filters, and what it does not#

Three scoping statements from the overview, each of which removes a class of expectation.

DNS Firewall only filters on the domain name. It does not resolve that name to an IP address to be blocked. Additionally, DNS Firewall filters DNS traffic, but it doesn't filter other application layer protocols, such as HTTPS, SSH, TLS, FTP, and so on.

A client with the address already cached, or hard-coded, or obtained from a resolver outside the VPC, is not affected. The firewall is a control on lookups through the VPC resolver, which is a specific and useful thing, and not a control on connections.

DNS queries originating within that VPC and passing through VPC DNS. DNS queries that pass through Resolver endpoints from on-premises resources into that same VPC that has DNS Firewall associated to its resolver.

Queries that arrive from on-premises through an inbound Resolver endpoint are filtered by the rule groups associated with the endpoint's VPC. A rule group written for a workload VPC is also, if the inbound endpoint lives there, filtering the corporate network's queries into AWS.

Resolver DNS Firewall is a Regional service, so objects that you create in one AWS Region are available only in that Region. To use the same rule group in more than one Region, you must create it in each Region.

The rule group you built is not protecting the VPC in the next Region, and it is not a copy away: it is a second rule group, with its own domain lists, to keep in step.

Rolling it out without the incident#

The documentation offers its own testing method, on the rule actions page:

When you first create a blocking rule, you can test it by configuring it with the action set to Alert. You can then look at the number of queries that the rule alerts on to see how many would be blocked if you set the action to Block.

For a walled garden, that means the block-everything rule goes in as ALERT first, and the query logs for a week tell you which names your workloads resolve that are not on the allow list. The CNAME targets will be in that list, because the firewall alerts on them exactly as it would block them. Promote to BLOCK when the alert count for legitimate traffic is zero.

Then the redirection setting is a decision rather than a default. In Terraform the attribute is firewall_domain_redirection_action, with values INSPECT_REDIRECTION_DOMAIN (the default) and TRUST_REDIRECTION_DOMAIN. Inspecting is safer and requires the allow list to name every hop, including the ones a vendor controls. Trusting is simpler and means an approved domain can be pointed at anything. There is no setting that gives you both; pick the one whose maintenance you are prepared for.

And remember the first association flips the VPC to fail-closed. If a SERVFAIL during a firewall impairment is worse for you than an unfiltered query, set FirewallFailOpen on the VPC's firewall configuration before the first rule group goes on, not after the first page.

What to do with it#

Resolve every allow-listed name and read the whole chain. dig +noall +answer app.example.com shows the CNAME hops. Each one is a name the firewall will evaluate under the default, and each one needs to be on a list with an action.

Treat the allow list as a dependency on other people's DNS. The CNAME targets belong to CDNs, SaaS vendors and load balancer services, and they change without telling you. Alert-mode logs are the detector; a scheduled resolve of the allow list is the earlier one.

Choose the block response for the responder, not the attacker. NXDOMAIN and NODATA are indistinguishable from an upstream problem to anyone on the client. If your own engineers will be debugging blocks, OVERRIDE to a name that says what happened saves a ticket.

Know the three failure shapes. NODATA or NXDOMAIN from a rule, SERVFAIL from fail-closed, success from fail-open. The query log fields firewall_rule_action and firewall_rule_group_id turn all of them into a sentence.

Set association priority deliberately when there is more than one rule group. Lowest number first, first match stops. Write the number down next to the reason.

Per Region, every time. A rule group is not a fleet-wide control until it exists in every Region with a VPC.

The through-line with the resolver rule that outranks the private zone and the custom DNS server that disabled private DNS: the VPC resolver's path has several places where the answer is decided by something other than the record, and the client only ever sees the answer. DNS Firewall adds one more such place, with a default that is strict in a direction most allow lists were not written to satisfy.