Skip to contents

public read access to a prefix of an S3 bucket

Source: R/s3.R
set_s3_public_access.Rd

Makes the files below a prefix of a bucket on Amazon S3, or on another S3 compatible service like MinIO, publicly readable, or private again, by adding or removing a statement in the bucket policy that grants everyone s3:GetObject on the objects below the prefix. Other statements of the policy are kept.

Usage

set_s3_public_access(path, public = TRUE)

Arguments

path

prefix in a bucket as s3://bucket/path, or s3://bucket for the whole bucket.

public

TRUE (default) to make the files below the prefix public, FALSE to remove the public access this function granted.

Value

(invisibly) the public URL of the prefix, ending in a slash.

Details

Whether a file is public is then decided by where it gets uploaded: files that upload_parquet() and data_to_parquet() write below the prefix are public, files elsewhere in the bucket stay private. Public files can be read by anyone at the URL returned, with parquet_tbl(), cache_parquet() and the other readers, and directories of parquet files are found through their manifests, since the policy does not allow listing the bucket.

This is the S3 counterpart of create_r2_bucket()'s public access, which R2 grants per bucket. Per-object ACLs are not used, they are disabled on buckets created since April 2023, and a public manifest listing private files would leak their names and metadata; keep public and private data under separate prefixes, a manifest of a directory above both, like one write_manifest() writes, stays private.

Requirements:

  • The credentials need the s3:GetBucketPolicy, s3:PutBucketPolicy and s3:DeleteBucketPolicy permissions.

  • Block Public Access has to allow public bucket policies, for the bucket and for the AWS account; the call stops when the bucket's own settings would keep it private.

Amazon S3 applies policy changes with a delay, for a few minutes after a call anonymous requests can still get 403 Forbidden now and then, or succeed after the access has been removed.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.