Skip to contents

write manifests for directories of parquet files on Cloudflare R2 or S3

Source: R/r2.R
write_manifest.Rd

Writes a _manifest.json file into directories on R2 or S3, listing the parquet files below the directory with their size, etag, modification time and footer metadata. For directories uploaded before, and to rebuild the manifest of a directory that fell behind.

Usage

write_manifest(path, purge_cache = TRUE)

Arguments

path

directory of (partitioned) parquet files as r2://bucket/path or s3://bucket/path, or a vector of these.

purge_cache

TRUE (default) or FALSE: purge the manifests from the Cloudflare cache of the bucket's custom domains. Ignored for S3.

Value

(invisibly) the manifests as r2://bucket/path/_manifest.json or the s3:// equivalent.

Details

upload_parquet() and data_to_parquet() write manifests for the directories they upload. A manifest falls behind after files were changed with other tools, after an upload or removal warned that it could not be updated, or after it was removed. Manifests let cache_parquet() and parquet_tbl() read a directory from its public URL, since directories can't be listed over https, and let clients other than R read what a dataset holds. On R2 the manifest gets purged from the Cloudflare cache of the bucket's custom domains like upload_parquet() does. write_manifest("r2://bucket") writes a manifest of the whole bucket at its root, which uploads and removals anywhere in the bucket then keep current.

The metadata comes from the parquet footers, as parquet_metadata() reads it. Files that an existing manifest lists with the same etag keep the metadata recorded there, the footers of all other files get read from the bucket, which costs a request or two per file, so writing the manifest of a large directory for the first time takes a moment. The manifest is written conditionally, like upload_parquet() writes them, and the call fails if it keeps changing underneath, e.g. because of uploads running at the same time.

Manifest format

A manifest is a JSON object with three members:

  • directory: true for the manifests in buckets, which list every parquet file below their directory.

  • metadata: an object with the metadata entries that all files share with identical values, the dataset's metadata, which is what parquet_metadata() returns for the directory. The entries are the ones data_to_parquet(metadata = ) wrote, columns included; the GeoParquet entry geo is left out.

  • files: an array with one object per parquet file, in the order of their keys, with

    • path: relative to the manifest's directory;

    • size: in bytes;

    • etag: for files uploaded by this package the MD5 hash of the file in hex, which is what R2 and S3 assign to single request uploads without encryption by KMS or customer keys;

    • modified: the modification time in UTC as YYYY-MM-DDTHH:MM:SSZ;

    • metadata: an object with the entries of the file's footer that differ from those of other files or that only this file has, left out if there are none.

The metadata of a file is the manifest's metadata with the file's own metadata added. So the column descriptions of a partitioned dataset sit in the manifest's metadata, and in a directory holding different datasets with each file. Values are stored as parquet_metadata() returns them, single values as strings and everything else as the JSON objects and arrays that data_to_parquet() wrote.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.