metadata of all parquet sources of a table — parquet

The metadata of every table from parquet_tbl() that a lazy table or collected data was read from, one row per table: for joins the left-hand table and the joined ones, for example to credit all of them in a caption. parquet_metadata() only has the left-hand table's.

Usage

parquet_sources(x, ...)

# S3 method for class 'tbl_lazy'
parquet_sources(
  x,
  from_files = FALSE,
  language = getOption("r2parquet.language"),
  ...
)

# S3 method for class 'data.frame'
parquet_sources(x, language = getOption("r2parquet.language"), ...)

# Default S3 method
parquet_sources(x, ...)

Arguments

x: a lazy DuckDB table, e.g. from parquet_tbl() and joins of such tables, or data collected with collect_parquet().
...: not used.
from_files: where lazy tables read the metadata of files on R2, S3 and at URLs from, see parquet_metadata().
language: language code like "fr" for the metadata of each source in that language, as collect_parquet() attaches it (attribution_fr becomes attribution, see its Languages section), or NULL for the metadata as stored. Defaults, like collect_parquet(), to the language the columns of a lazy table were selected in by parquet_tbl() or select_language(), else to the r2parquet.language option. Without a language, a source with text in several languages and no language entry has no plain attribution to return, which a message points out once per session.

Value

a tibble with one row per source table, left-hand table first: path, the path(s) passed to parquet_tbl(), separated by ", "; a column for each metadata entry that holds a single value in any of the sources, like title, attribution or licence, NA for sources without it; and metadata, the whole metadata of each source as returned by parquet_metadata().

Details

The tables get found through the query of the lazy table, so tables joined with dplyr::left_join() and the other joins, dplyr::semi_join(), dplyr::union_all() and the like count, also when they were never assigned to a variable. collect_parquet() attaches the result to the collected data as the parquet_sources attribute. A table read from several paths in one parquet_tbl() call is one row, with its metadata merged as parquet_metadata() merges it. Tables not created by parquet_tbl() in the current session, like data copied to the connection, have no row.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

R2_ACCOUNT_ID: the R2 account id.
R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.
R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.
R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.
AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.
AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).
R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.

Examples

# \donttest{
if (requireNamespace("curl", quietly = TRUE) && curl::has_internet()) {
  census <- "https://data.mountainmath.ca/census/2021/"
  # the dissemination blocks of one dissemination area, with their population
  blocks <- parquet_tbl(paste0(census,"ldb_000b21f_e.parquet")) |>
    dplyr::filter(DAUID == "59150307") |>
    dplyr::select(DBUID,geometry)
  population <- parquet_tbl(paste0(census,"2021_92-151_X.parquet")) |>
    dplyr::select(DBUID,DBPOP2021)
  data <- collect_parquet(dplyr::inner_join(blocks,population,by="DBUID"))
  paste(unique(parquet_sources(data)$attribution),collapse=" | ")
}
#> duckdb keeps downloaded extensions and secrets in a temporary directory:
#> ℹ /tmp/RtmprWVhal/duckdb
#> This is removed when the R session ends.
#> • Extensions are re-downloaded each session.
#> • Secrets are lost.
#> ℹ Run duckdb(shared_home = TRUE) (or create ~/.duckdb) to keep them (suitable for most users).
#> ℹ Run duckdb(shared_home = FALSE) to accept the temporary directory (and silence this message).
#> ℹ See ?duckdb_storage for details and alternatives.
#> The data carries its text in en and fr with no plain text to fall back on, so no labels or attributions are resolved without a language. Set options(r2parquet.language = "en") for the session, or pass language = "en" to collect_parquet(), parquet_sources() or parquet_sources_attribution(); parquet_metadata() returns the metadata in all languages. This message is shown once per session.
#> Warning: Unknown or uninitialised column: `attribution`.
#> [1] ""
# }