An S3 endpoint policy that names only your buckets breaks dnf and docker pull

The tightening is sensible. A private subnet reaches S3 through a gateway endpoint, the endpoint policy is the default full-access one, and a review asks why instances in this VPC can read any bucket in any account. The answer is to scope the policy to the buckets the workload uses. The change is a few lines, it is applied, and the workload keeps reading its buckets.

Then dnf update fails. Then the container hosts cannot pull images. Both errors name S3, and neither names a bucket that is yours.

Where the repositories actually are#

From AWS's guidance on updating Amazon Linux without internet access:

Amazon Linux repositories are hosted in Amazon Simple Storage Service (Amazon S3) buckets. To update and install packages on your instance without an internet connection, create an Amazon Virtual Private Cloud (Amazon VPC) gateway endpoint for Amazon S3. Include a policy in the VPC endpoint that allows access to the repository buckets.

The buckets have names, and the names are Regional. For Amazon Linux 2023 in us-east-1, the same page gives the policy:

{
  "Statement": [{
    "Principal": "*",
    "Action": ["s3:GetObject"],
    "Effect": "Allow",
    "Resource": ["arn:aws:s3:::al2023-repos-us-east-1-de612dc2/*"]
  }]
}

For Amazon Linux 2 the buckets are amazonlinux.us-east-1.amazonaws.com and amazonlinux-2-repos-us-east-1. Nothing about these appears in your account. The instance's package manager resolves a repository URL, the URL is an S3 endpoint, the gateway endpoint carries the request, and the endpoint policy evaluates s3:GetObject on a bucket that is not in your allow list. Denied.

A policy that lists arn:aws:s3:::my-app-data/* and nothing else has, as a side effect, disabled package installation on every Amazon Linux host in the subnet. The error says access denied on al2023-repos-us-east-1-de612dc2, which is accurate and unhelpful unless you already know what that bucket is.

ECR has the same shape#

Pulling a container image from ECR is two services. The ECR interface endpoint documentation:

The gateway endpoint is required because Amazon ECR uses Amazon S3 to store your image layers. When your containers download images from Amazon ECR, they must access Amazon ECR to get the image manifest and then Amazon S3 to download the actual image layers. The following is the Amazon Resource Name (ARN) of the Amazon S3 bucket containing the layers for each Docker image.

arn:aws:s3:::prod-region-starport-layer-bucket/*

So the ECR endpoints can be perfectly configured, with their own policies allowing exactly the pull role, and the pull still fails at the layer download because the S3 endpoint policy does not allow s3:GetObject on prod-us-east-1-starport-layer-bucket. The manifest arrives; the layers do not. AWS gives the minimum S3 statement for this on the same page, and it is the one that has to be merged into whatever bucket list your review produced:

{
  "Sid": "Access-to-specific-bucket-only",
  "Principal": "*",
  "Action": ["s3:GetObject"],
  "Effect": "Allow",
  "Resource": ["arn:aws:s3:::prod-us-east-1-starport-layer-bucket/*"]
}

The pattern generalises. Any AWS service that stores what it hands you in S3, and hands it to you by redirecting you to S3, has a bucket name that your endpoint policy has to know about. The two above are the ones that break the base operating system and the container runtime, which is why they surface first.

Three traps in the policy language#

The endpoint policy documentation lists considerations that read as footnotes until one of them is the reason a policy does not do what it says.

Gateway endpoints need Principal: *.

With gateway endpoints, the Principal element must be set to *. To specify a principal, use the aws:PrincipalArn condition key.

And the consequence of doing it the way an IAM-trained hand wants to:

If you specify the principal in the following format, access is granted to the AWS account root user only, not all users and roles for the account.

"AWS": "account_id"

A gateway endpoint policy with "Principal": {"AWS": "123456789012"} is not "anyone in my account". It is the root user, and nothing else in the account can use the endpoint. The policy validates, applies, and denies every role.

Wildcards do not work against system identifiers.

You can't use wildcard characters (* or ?) or numeric condition operators with global context keys that reference system-generated identifiers (for example, aws:PrincipalAccount or aws:SourceVpc).

When you use a string condition operator, you must use at least six consecutive characters before or after each wildcard character.

A condition like "aws:PrincipalArn": "arn:aws:iam::*:role/deploy-*" has a wildcard in the account position, and a bucket resource of arn:aws:s3:::app-* has only four characters before its wildcard. Neither does what it looks like it does.

A service that does not support endpoint policies gets full access regardless.

Not all AWS services support endpoint policies. If an AWS service doesn't support endpoint policies, we allow full access to any endpoint for the service.

When you create a VPC endpoint for an endpoint service other than an AWS service, we allow full access to the endpoint.

The policy on an endpoint for a service in the first group is decorative, and an endpoint to a partner or your own PrivateLink service has no policy at all. A control that is enforced on S3 and ignored on another endpoint in the same VPC is a control whose coverage has to be checked per service.

It is one of three policies, and the other two can lock you out#

The documentation is careful about scope:

An endpoint policy does not override or replace identity-based policies or resource-based policies.

A request through an S3 endpoint has to pass the endpoint policy, the caller's IAM policy, and the bucket policy. Tightening any one of them tightens the result. The bucket policy is where the other common mistake lives, because the natural pairing of "endpoint policy allows only my buckets" is "bucket policy allows only my endpoint", and AWS's bucket policy examples carry a warning at the top of the page:

Bucket permissions that are intended to specifically limit bucket access to connections originating from your VPC endpoint can block all connections to the bucket.

And under each example, the specific way it happens:

This policy disables console access to the specified bucket because console requests don't originate from the specified VPC endpoint.

A Deny with StringNotEquals aws:SourceVpce denies the AWS console, the CLI from a laptop, and any Lambda outside that VPC. If the deny is written before the allow for an administrator path exists, the person applying it has just removed their own access to the bucket, including the access they would need to fix the policy. AWS's aws:SourceVpc variant has the same effect for anything outside the VPC.

One more scoping detail from the S3 endpoint documentation, for anyone counting on the endpoint policy as the sole path control: if the gateway endpoint is IPv4-only and the client resolves a dual-stack S3 hostname to an AAAA record, that traffic does not use the gateway endpoint and "will be dropped or routed over an IPv6-compatible path if one is present" — through the internet gateway, if the VPC has one. The endpoint policy never sees it. The documented answer is the bucket policy with the aws:SourceVpce condition, which is the control that evaluates regardless of which path the request took.

A policy that survives the review#

The shape that scopes without breaking the host:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WorkloadBuckets",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-app-data",
        "arn:aws:s3:::my-app-data/*"
      ],
      "Condition": {
        "StringEquals": { "aws:PrincipalAccount": "123456789012" }
      }
    },
    {
      "Sid": "AmazonLinuxRepositories",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::al2023-repos-us-east-1-de612dc2/*"
    },
    {
      "Sid": "EcrImageLayers",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::prod-us-east-1-starport-layer-bucket/*"
    }
  ]
}

Principal is * throughout because this is a gateway endpoint and the documentation requires it; the account restriction is a condition, not a principal. The AWS buckets are read-only and unconditioned, because the request is made by the OS or the container runtime using whatever credentials it has, and because the objects in them are AWS's. The Region is in both bucket names and changes with the endpoint's Region.

The documentation adds that "after you update an endpoint policy, it can take a few minutes for the changes to take effect", so a test immediately after terraform apply can report the old behaviour.

What to do with it#

Before scoping an S3 endpoint policy, list what the hosts fetch from S3 that is not yours. Package repositories and container layers are the two every fleet has. Anything else that redirects you to an S3 URL is a candidate.

Read the access-denied bucket name as a dependency, not an intruder. al2023-repos-*, amazonlinux-2-repos-* and *-starport-layer-bucket are AWS infrastructure. The fix is an allow statement, not an investigation.

On a gateway endpoint, Principal is * and the account goes in a condition. The account-ID principal form means the root user only, and the policy will not tell you.

No wildcards against aws:PrincipalAccount or aws:SourceVpc, and six characters either side of any wildcard you do use. Both are stated limits, and a policy that breaks them is accepted and wrong.

Check which of your endpoints honour a policy at all. Non-supporting AWS services and every non-AWS endpoint service are full access, whatever the policy document says.

Write the bucket-policy deny last, and keep a path that is not the endpoint. The console is not in your VPC. Neither is the engineer who will be fixing this at two in the morning.

The shape is the one from interface endpoints with private DNS off and the NAT Gateway that bills the download: a path to S3 that the workload uses without the workload's owner knowing it is S3. The endpoint policy is the first control that makes those hidden paths visible, by breaking them.