Skip to contents

collect a lazy DuckDB table of parquet data with its metadata

Source: R/geoparquet.R
collect_parquet.Rd

Executes the query and returns the result as a tibble, or as an sf object if the table has GEOMETRY columns, with the metadata of the parquet files attached.

Usage

collect_parquet(
  tbl,
  attach_metadata = TRUE,
  language = getOption("r2parquet.language")
)

Arguments

tbl

lazy DuckDB table, e.g. from parquet_tbl() or sf::st_filter().

attach_metadata

TRUE (default) or FALSE: attach the metadata of the parquet files as attributes.

language

language code like "en" or "fr" to collect the labels and columns in, see the Languages section, or NULL for the dataset as stored. Defaults to the language the table's columns were selected in by parquet_tbl() or select_language(), else to the r2parquet.language option.

Value

a tibble, or an sf object if the table has GEOMETRY columns.

Details

GEOMETRY columns, like those of GeoParquet files written by data_to_parquet(), become sf geometry columns with their crs. The active geometry column is the one named geometry, as data_to_parquet() writes it, or else the first GEOMETRY column, use sf::st_set_geometry() to switch to another one. Missing (NULL) geometries, e.g. from a left join, become empty geometries, since sf has no missing geometries.

Reading the files fails with the errors described in the Errors section of parquet_tbl(), e.g. an r2parquet_network_error when the server can't be reached.

Parquet is stored by column, so a query that keeps only the columns it needs downloads only those. Collecting a table on R2, S3 or at a URL that still has every column it was read with (nothing selected away, no summary) gives a message, once per session, with a select() of the columns it is about to fetch, to paste into the query and prune.

For tables from parquet_tbl() the metadata of the parquet files (see parquet_metadata()) gets attached to the result: the whole list as the metadata attribute, and the label entries of columns as label attributes of the columns, as the haven and labelled packages use them. The files are looked up through the table, so the metadata follows the table through dplyr verbs and is the metadata of the left-hand table of joins. The metadata of every table from parquet_tbl() the result was read from, joined tables included, gets attached as the parquet_sources attribute, see parquet_sources(), and columns the left-hand table has no label for get the labels of the other tables.

Languages

Datasets can carry their text in several languages, see the Languages section of data_to_parquet() for the convention: labels and columns per language are named with the language as suffix, label_fr and name_fr, next to a plain label and name or without them, the language entry names the language of the plain ones, and which suffixes are languages is declared by the metadata (the languages and language entries), so a column like rate_men next to rate is left alone. With language, the result is in that language where the data has it:

  • column labels come from the label_<language> entries, else from label, else from the label in the dataset's language;

  • a column in that language takes the place and name of its plain column, name_fr gets collected as name, and the columns in the other languages get left out. The plain column stays where the data has none in the language asked for, and a suffixed column without a plain column or a peer in another language is left alone. The selection happens before the query runs, so the columns left out don't get read.

The metadata attached (the metadata and parquet_sources attributes) describes the result in that language too: title_fr, attribution_fr and label_fr take the place of title, attribution and label (falling back to the plain entry and then to the dataset language's), the other languages' entries and languages are left out, columns is keyed by the columns as collected and the language entry names the language delivered. Writing the result with that metadata therefore gives a correct dataset in one language. The metadata as stored, with all languages, is parquet_metadata(tbl) on the lazy table or the path. A dataset that doesn't declare the language asked for is collected as stored, metadata included.

Without language the result is the dataset as stored, with all its columns, labelled with the plain labels, else those in the dataset's language; columns with labels only in other languages get none. A dataset that carries its text in several languages without a language entry, as Statistics Canada's English and French data does, therefore gets no labels and no attribution until a language is asked for, which a message points out once per session. The default is the r2parquet.language option, so options(r2parquet.language = "fr") sets the language once for a session.

With the option set, parquet_tbl() already selects the columns in that language, through select_language(), so the lazy table has name rather than name_en and name_fr and queries can use it; the labels then come from the stored column it stands in for, and a table whose columns were selected in a language is described in that language when none is passed here, whatever the option says.