Skip to contents

metadata of parquet files

Source: R/metadata.R
parquet_metadata.Rd

Reads the key-value metadata from the footer of parquet files, like the source, license, notes and columns entries written by data_to_parquet() with its metadata argument. Only metadata gets read, no data.

Usage

parquet_metadata(x, ...)

# S3 method for class 'character'
parquet_metadata(x, from_files = FALSE, language = NULL, ...)

# S3 method for class 'tbl_lazy'
parquet_metadata(x, from_files = FALSE, language = NULL, ...)

# S3 method for class 'data.frame'
parquet_metadata(x, language = NULL, ...)

# Default S3 method
parquet_metadata(x, ...)

Arguments

x

what to read the metadata of:

  • URL, r2://bucket/path, s3://bucket/path or local path of a parquet file or of a directory of (partitioned) parquet files, or a vector of these;

  • a lazy DuckDB table from parquet_tbl();

  • data collected with collect_parquet(), which carries the metadata in its metadata attribute.

...

not used.

from_files

where to read the metadata of files on R2, S3 and at URLs from:

  • FALSE (default): the manifest, where there is one.

  • TRUE: the footers of the parquet files, current even when the files were changed with other tools.

language

NULL (default) for the metadata as stored, with the entries of all languages, or a language code like "fr" for the metadata as collect_parquet() attaches it when collecting in that language, see its Languages section: title_fr and label_fr take the place of title and label, the other languages' entries are left out, columns follows the columns collected, the language entry names the language and languages is left out. Unlike collect_parquet(), the r2parquet.language option does not apply, so that parquet_metadata(tbl) always returns everything the files hold.

Value

a named list of the metadata entries, empty if the files have none (or the data was collected without attaching it), of class parquet_metadata, which only changes how it prints: one line per entry, lists as compact JSON, and the columns entry as a table of the columns with their labels and descriptions. Entries in several languages (title and title_fr, label_en and label_fr) are consolidated into one row with a column per language, the dataset's language (its language entry, or plain text without one) first.

Details

Entries are strings in the footer, entries holding JSON objects or arrays, like columns, get parsed into lists. The files of a partitioned directory get written together and should have the same metadata; where they disagree the values of the first file are used, with a warning. The geo entry with the GeoParquet metadata is left out, the crs of geometry columns is part of the GEOMETRY column type and comes with the geometry when collecting via collect_parquet().

For paths on R2 or S3 and at URLs the metadata comes from the manifest (_manifest.json, see write_manifest()) of the directory, or of the nearest directory above the path that has one, which records the footer metadata of every parquet file below it: one request for a whole dataset instead of a footer read per file. upload_parquet() and data_to_parquet() keep the manifests current, files changed with other tools are not reflected until write_manifest() rebuilds the manifest. The footers of the files get read instead with from_files = TRUE, and for paths without a manifest (or with one written before manifests recorded metadata), presigned URLs, local paths and tables that read local copies from the cache.

A lazy table reads its metadata once and keeps it, for collect_parquet(), parquet_sources() and the language selection. data_to_parquet(), upload_parquet(), remove_parquet(), write_manifest(), cache_parquet() and remove_parquet_from_cache() make the tables reading the paths they change read it again, so a table whose files were rewritten in place with data_to_parquet() describes the new files; files changed outside this session are read again with from_files = TRUE.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.