Skip to content
Networking6 min read

us-east-1a is not the same place in your account and theirs

Availability Zone names are mapped per account, deliberately and randomly, in every Region that existed before November 2012. A PrivateLink consumer who builds in us-east-1a can be refused an endpoint for a service that is, by the provider's naming, in us-east-1a. The identifier that is the same everywhere is the AZ ID, and almost nothing is written in it.

· Venerable Networks

A partner publishes a service over PrivateLink. They tell you it is deployed in us-east-1a and us-east-1b. You create the interface endpoint in your us-east-1a subnet and the API refuses it:

The VPC endpoint service com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0
does not support the Availability Zone of the subnet: subnet-0abc...

Both of you are telling the truth. The service is in their us-east-1a. Your subnet is in your us-east-1a. Those are two different buildings.

The mapping is per account, on purpose

From the AZ IDs documentation:

Originally, we decided to independently map Availability Zones to codes in each AWS account. This ensures that resources are distributed across the Availability Zones for these Regions, even if most customers chose the first Availability Zone in the Region. For example, the us-east-1a for your AWS account might not be the same physical location as the us-east-1a for another AWS account.

The reason is load spreading. If everyone picked the first zone in the list, the first zone would be the busiest, so AWS shuffles the letters per account. The RAM documentation is blunter about the method: "AWS maps the physical Availability Zones randomly to the Availability Zone names for each AWS account."

Within one account this is invisible and harmless. The letter is a stable label for a place and you never need to know which place. It becomes a problem the moment two accounts have to agree on a location, which is exactly what PrivateLink, shared subnets, and anything cross-account built on zonal placement require.

It is the old Regions, which is to say the busy ones

The same page carries a detail that decides whether this affects you:

As AWS learned more about customer usage patterns, and as AWS evolved, we determined that it was not necessary to continue to independently map Availability Zones to codes. All Regions introduced after November 2012 use a uniform mapping of Availability Zones to codes.

So in a Region launched after late 2012, eu-west-2a is the same place in every account and this post is moot. In us-east-1, us-west-2, eu-west-1, ap-southeast-1, ap-northeast-1 and the other early Regions — the ones with the most accounts, the most PrivateLink services, and the most cross-account architecture — the names are shuffled. The failure concentrates precisely where the most people are building.

The identifier that is the same everywhere

AWS's fix is a second name for every zone that does not vary:

To coordinate Availability Zones across accounts in all Regions, even those that independently map Availability Zones, use the AZ IDs, which are unique and consistent identifiers for Availability Zones. For example, use1-az1 is an AZ ID for the us-east-1 Region, and it has the same physical location in every AWS account.

Your account's mapping is one call:

aws ec2 describe-availability-zones --region us-east-1 \
  --query 'AvailabilityZones[].[ZoneName,ZoneId]' --output table
# ----------------------------
# |  us-east-1a |  use1-az6  |
# |  us-east-1b |  use1-az1  |
# |  us-east-1c |  use1-az2  |
# |  us-east-1d |  use1-az4  |
# |  us-east-1e |  use1-az3  |
# |  us-east-1f |  use1-az5  |
# ----------------------------

Run the same command in the partner's account and the right-hand column is the same set of six IDs in a different order against the letters. The IDs are the ground truth. The letters are each account's private alias for them.

An endpoint service's Network Load Balancer is in specific zones. A consumer can create an interface endpoint only in zones the service is in, and the comparison is done on physical zones — on AZ IDs — not on letters. The PrivateLink documentation for endpoint services states the consequence directly:

When service consumers retrieve information about an endpoint service, they can see only the Availability Zones that they have in common with the service provider. When the service provider and service consumer are in different accounts, an Availability Zone name, such as us-east-1a, might be mapped to a different physical Availability Zone in each AWS account. You can use AZ IDs to consistently identify the Availability Zones for your service.

So describe-vpc-endpoint-services from the consumer side returns zone names — but they are the consumer's names for the zones the service is physically in. If the partner's NLB is in their us-east-1a and us-east-1b, and those are use1-az6 and use1-az1, your describe-vpc-endpoint-services shows you whichever of your letters map to use1-az6 and use1-az1. Which may well be us-east-1a and us-east-1b, or may be us-east-1c and us-east-1f. The letters the partner gave you are not information you can use.

aws ec2 describe-vpc-endpoint-services \
  --service-names com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0 \
  --query 'ServiceDetails[0].AvailabilityZones'
# [ "us-east-1c", "us-east-1f" ]

That is the authoritative answer for your account, in your letters. Build the endpoint in those.

The cost of a partial match

AWS's SaaS anti-patterns guidance works through what happens when provider and consumer overlap on some zones but not all:

Assume the SaaS offering is deployed in use1-az1 and use1-az2, but the consumer is using all three Availability Zones, including use1-az3. The interface VPC endpoints are deployed on the consumer side in use1-az1 and use1-az2, and now the application in use1-az3 needs to access one of these endpoints.

The consumer's use1-az3 workloads have no local endpoint. They reach one in another zone, paying cross-AZ data transfer for every request and losing zonal isolation for that dependency. The guidance follows the arithmetic through to a 67/33 split landing on the provider's two zones, and the recommendation is the obvious one: a provider should deploy in every zone of the Region, because consumers will be in every zone and a consumer cannot put an endpoint where the provider is not.

Notice the guidance is written in AZ IDs. That is not a style choice.

What to do with it

Communicate zones as AZ IDs, always. When a provider tells a consumer where a service is, or an architecture document records which zones a shared subnet spans, the letter is wrong in every account but the author's. use1-az1 is right in all of them. The habit costs nothing and removes the whole failure.

Resolve the mapping before placing anything cross-account. One describe-availability-zones per account involved, kept next to the design. If a Terraform module takes zone names as input and is applied in more than one account, it is choosing a different physical zone in each one, and that is sometimes exactly what you did not want.

Read describe-vpc-endpoint-services rather than the partner's email. It is the only thing that tells you which of your subnets can host the endpoint, and it is already translated into your account's letters.

Providers: deploy in every zone, or publish the AZ IDs you are in. Ideally both. A consumer in a zone you skipped has no good option, and a consumer who only has your letters has no information.

Check the Region's age. If it launched after November 2012, the mapping is uniform and the letters are safe. If it is one of the originals, assume nothing.

The pattern is a familiar one on this site: a label that is stable where you can see it and means something different one account over. The Transit Gateway's route tables do this with "the route table". The HOME_NET variable in Lab 19 does it with "the home network". Here it is "zone a", and the fix is the same shape in all three — find the identifier that does not move, and write in that.