Skip to contents

transfer files to Cloudflare R2 or S3

Source: R/r2.R
upload_parquet.Rd

Uploads a single file, or all parquet files in a directory of (partitioned) parquet files keeping the directory structure, to a bucket that has to exist (see create_r2_bucket()). Directory uploads write a manifest (_manifest.json), and all uploads keep the existing manifests of the dataset current.

Usage

upload_parquet(path, destination, purge_cache = TRUE, remove_stale = FALSE)

Arguments

path

local file, or directory of (partitioned) parquet files.

destination

destination as r2://bucket/path or s3://bucket/path:

  • for a file: the path of the file, or with a trailing slash (/), or just the bucket, the directory it goes into under its basename;

  • for a directory: the directory its content goes into, which has to be a directory in the bucket, not the bucket itself.

purge_cache

TRUE (default) or FALSE: purge the uploaded files from the Cloudflare cache of the bucket's custom domains. Ignored for S3.

remove_stale

what happens to parquet files below destination that are not in the uploaded directory:

  • FALSE (default): they are left in place.

  • TRUE: they get removed before the upload, so that the destination mirrors the directory, e.g. when a dataset got rebuilt without a partition or file. Other files in the directory and files above destination are never touched; manifests of directories below destination lose the removed files, and go when nothing is left.

Value

(invisibly) the uploaded files as r2://bucket/path or s3://bucket/path.

Details

Each file goes up in a single request and is read into memory for that, so a file has to fit into memory, and the 5 GB limit of single requests applies; split larger data into partitions with data_to_parquet(). File names with *, ? or [ are refused, since DuckDB reads them as patterns when the files are queried. If an R2 bucket is served via a custom domain, the uploaded files get purged from the Cloudflare cache, see the Credentials section.

A single file can be of any type, like a JSON index or a README that gets published next to a dataset. It goes up with the content type for its extension, as mime::guess_type() guesses it (application/json for .json, text/csv for .csv, and so on), so that browsers and other clients handle it correctly; parquet files and unknown extensions are sent as application/octet-stream. remove_parquet() removes such files again. _manifest.json files are reserved for manifests and can't be uploaded, use write_manifest() to write them.

Manifests

A manifest lists the parquet files below its directory with their size, etag, modification time and footer metadata, see write_manifest() for the format. It lets cache_parquet() and parquet_tbl() read a directory from a public URL, where the files can't be listed, and lets other clients read the dataset's metadata without the parquet footers.

  • The manifest of the destination of a directory upload is built from a listing of the directory, so it also repairs a manifest that fell behind.

  • Since a manifest lists all parquet files below its directory, the existing manifests of the directories above the destination get updated too, and for directory uploads those below it, so that uploading a single file or a new partition into a dataset keeps its manifest current. These get patched from what they record, without listing their directories: the uploaded and removed files are swapped in and out, and the metadata of the uploaded files is read from the local files. write_manifest() rebuilds a manifest from a listing when files were changed by other means.

  • Manifests are written conditionally, so that an upload running at the same time can't overwrite an update, and the update starts over from the current manifest when that happens. The upload warns, after the files are up, when a manifest could not be updated this way.

  • Uploading a single file never creates a manifest. A directory can't be uploaded to the root of the bucket, since its manifest would list the whole bucket and remove_stale would empty it; write_manifest() writes a manifest of a whole bucket on request.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.