Skip to contents

local copies of parquet files on R2, S3 or at URLs

Source: R/cache.R
cache_parquet.Rd

Downloads parquet files, or directories of (partitioned) parquet files, to a local cache directory and keeps them current, so that data used regularly can be queried locally instead of over the network. Once a dataset is cached, parquet_tbl() reads the copy by default as long as it is current, see its cache argument.

Usage

cache_parquet(path)

Arguments

path

URL, r2://bucket/path or s3://bucket/path of a parquet file or a directory of (partitioned) parquet files, or a vector of these.

Value

the local paths of the cached files or directories, one per path, ready for parquet_tbl().

Details

Each call checks the remote files and only downloads files whose etag changed or that are new, removes files from the cache that are gone remotely, and records the etags in a manifest (_manifest.json) in the cache directory. If the remote can't be reached, e.g. offline, an existing cached copy gets used with a warning. Local paths are returned as they are.

  • r2:// and s3:// paths get checked against a listing of the bucket.

  • Files at URLs, including presigned URLs, get checked with a request for their first byte, which returns the etag.

  • Directories at URLs can't be listed, they need a manifest next to the files, which upload_parquet() and data_to_parquet() write for directories uploaded to R2 or S3 and write_manifest() writes for existing ones.

The cache directory is the one in the R2PARQUET_CACHE_PATH environment variable, or the user cache directory (tools::R_user_dir("r2parquet", "cache")) if that is not set. Set the environment variable, e.g. in .Renviron, for a cache local to a project. The cache mirrors the remote layout, with bucket names only unique within an account or service:

  • r2://bucket/path as r2/<account id>/bucket/path;

  • s3://bucket/path as s3/aws/bucket/path on Amazon S3, and as s3/<host>/bucket/path (<host>_<port> with a port) for a custom endpoint;

  • URLs as <scheme>/<host>/<path>. The package manages the files below the cache directory, files it does not know about get removed when a directory is synced. Downloads go through a temporary file, and files uploaded by this package get checked against their etag, which is the MD5 hash of the content, so an interrupted download never ends up in the cache.

list_parquet_cache() lists what is in the cache, list_parquet() on the remote paths tells whether the copies are current, and remove_parquet_from_cache() removes copies.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.