The package focuses on parquet because it is uniquely suited to hold large static dataasets that can be read strategically instead of downloading the entire file, although how effective it is at that at least partially depends on how well the choices made in ordering, partitioning and row groups when uploading the file suit the use case of the person consuming the data.
Next to the data a parquet file carries a schema, statistics that let
readers skip the parts they don’t need, and free-form key-value
metadata. r2parquet uses that metadata to add important
context, where the data comes from, under which licence, and what its
columns mean, and it copies the description into a manifest next to the
files on R2, where clients can read it without opening a single parquet
file. This article walks through where these pieces live, for a single
file, a directory holding several datasets and a partitioned dataset,
using the public bucket behind
https://data.mountainmath.ca.
What is in a parquet file
A parquet file stores its rows in row groups, and within a
row group each column separately. For every column of every row group
the file records statistics, the minimum and maximum say, so a reader
that filters on a column only downloads the row groups whose range
matches. data_to_parquet() sorts the rows by the
sort_by columns so that these ranges are narrow, and
row_group_size sets how many rows a row group holds, the
trade-off being between selective filters and the size of the file
metadata.
That metadata sits in the footer at the end of the file: the
schema, the statistics and the location of every column chunk, and
key-value metadata, arbitrary strings that tools can use as
they like. Reading the footer is all parquet_tbl() does
when it opens a remote file, a request for the last few kilobytes.
Take the file with the 2021 census dissemination block boundaries in the public bucket, the one the Accessing public buckets article queries.
db_url <- "https://data.mountainmath.ca/census/2021/ldb_000b21f_e.parquet"
db_table <- parquet_tbl(db_url)DuckDB exposes the footer directly, here the number of rows and row groups of the 330 MB file, which is all that has been read so far.
con <- parquet_connection()
DBI::dbGetQuery(con, paste0(
"SELECT num_rows, num_row_groups FROM parquet_file_metadata(",
DBI::dbQuoteString(con, db_url), ")"))
#> num_rows num_row_groups
#> 1 498547 244The key-value metadata holds whatever
data_to_parquet(metadata = ) was given when the file was
written, single values as they are and lists as JSON, plus for
GeoParquet files the geo entry with the GeoParquet
metadata.
DBI::dbGetQuery(con, paste0(
"SELECT decode(key) AS key, left(decode(value), 60) AS value FROM parquet_kv_metadata(",
DBI::dbQuoteString(con, db_url), ")"))
#> key
#> 1 title_en
#> 2 title_fr
#> 3 source_url_en
#> 4 source_url_fr
#> 5 reference_guide_en
#> 6 reference_guide_fr
#> 7 licence_en
#> 8 licence_fr
#> 9 licence_url_en
#> 10 licence_url_fr
#> 11 languages
#> 12 attribution_en
#> 13 attribution_fr
#> 14 notes_en
#> 15 notes_fr
#> 16 columns
#> 17 primary_key
#> 18 geometry_repairs
#> 19 processing
#> 20 geo
#> value
#> 1 2021 Census – Boundary files – Dissemination blocks
#> 2 Recensement de 2021 – Fichiers des limites – Îlots de diffus
#> 3 https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-
#> 4 https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-
#> 5 https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/
#> 6 https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/
#> 7 Statistics Canada Open Licence
#> 8 Licence ouverte de Statistique Canada
#> 9 https://www.statcan.gc.ca/en/terms-conditions/open-licence
#> 10 https://www.statcan.gc.ca/fr/reference/licence
#> 11 ["en","fr"]
#> 12 Statistics Canada, 2021 Census dissemination block boundary
#> 13 Statistique Canada, Fichier des limites des îlots de diffusi
#> 14 The identifiers of the containing geographies come from the
#> 15 Les identificateurs des géographies englobantes proviennent
#> 16 {"DBUID":{"label_en":"Dissemination block ID","label_fr":"Id
#> 17 ["DBUID"]
#> 18 {"columns":["DBUID"],"rows":[["10010273010"],["10010667008"]
#> 19 Repaired 830 of 498,547 geometries that were invalid, via sf
#> 20 {"version":"2.0.0","primary_column":"geometry","columns":{"gparquet_metadata() reads these entries back as a list,
parsing the JSON ones and leaving out geo: the crs of a
geometry column lives in the column’s type, which is where
collect_parquet() takes it from.
from_files = TRUE makes it read the footer of the file, we
will see below what it reads otherwise. The list prints one line per
entry, cut to the width of the console, with lists like the 830
geometries that were repaired when the file was written shown as compact
JSON.
db_metadata <- parquet_metadata(db_url, from_files = TRUE)
db_metadata
#> <parquet metadata: 19 entries>
#> en fr
#> title 2021 Census – Boundary files … Recensement de 2021 – Fichier…
#> source_url https://www12.statcan.gc.ca/c… https://www12.statcan.gc.ca/c…
#> reference_guide https://www12.statcan.gc.ca/c… https://www12.statcan.gc.ca/c…
#> licence Statistics Canada Open Licence Licence ouverte de Statistiqu…
#> licence_url https://www.statcan.gc.ca/en/… https://www.statcan.gc.ca/fr/…
#> languages en, fr
#> attribution Statistics Canada, 2021 Censu… Statistique Canada, Fichier d…
#> notes The identifiers of the contai… Les identificateurs des géogr…
#> columns <13 columns, see below>
#> primary_key DBUID
#> geometry_repairs {"columns":"DBUID","rows":["10010273010","10010667008","10010…
#> processing Repaired 830 of 498,547 geometries that were invalid, via sf:…
#>
#> columns en fr
#> DBUID Dissemination block ID Identificateur unique de l'îlot …
#> DAUID Dissemination area ID Identificateur unique de l'aire …
#> CTUID Census tract ID Identificateur unique du secteur…
#> CSDUID Census subdivision ID Identificateur unique de la subd…
#> CMAUID Census metropolitan area or cens… Identificateur unique de la régi…
#> CMAPUID CMA/CA provincial part ID: Provi… Identificateur unique de la part…
#> CDUID Census division ID Identificateur unique de la divi…
#> PRUID Province or territory ID Identificateur unique de la prov…
#> DGUID Dissemination geography unique i… Identificateur unique de géograp…
#> DBRPLAMX Representative point, x (Lambert) Point représentatif, x (Lambert)
#> DBRPLAMY Representative point, y (Lambert) Point représentatif, y (Lambert)
#> LANDAREA Land area (km²) Superficie des terres (km²)
#> geometry Boundary, Statistics Canada Lamb… Limites, projection conique conf…The entries are free-form, the ones in this bucket describe
provenance and licence. Metadata can be multilingual, see Languages below. A columns entry, a
list with one entry per column holding a label and a description, is the
one entry collect_parquet() treats specially, the labels
get attached to the collected columns as label attributes.
Column labels can be seen in action in the partitioned dataset below.
Three more entries have a fixed meaning: primary_key names
the columns that together identify the rows, which
data_to_parquet() checks before writing, one column for the
files here and several for long tables like the partitioned dataset
below, where rows are identified by area and characteristic.
processing and geometry_repairs record
geometries that data_to_parquet() repaired, how many and
with which library, and which rows by their primary key.
Metadata in the footer is part of the file, so changing it means writing the file again.
Languages
Metadata can be multilingual, with any number of languages. To enable
multiple language support the languages need to be declared in the
metadata languages field, e.g. c("en", "fr")
to declare English and French labels. Then fields that carry one of
these language suffixes, appended with underscore,
e.g. title_en and title_fr,
label_en and label_fr in the column entries,
are interpreted as multilingual labels by the package.
db_metadata |> names()
#> [1] "title_en" "title_fr" "source_url_en"
#> [4] "source_url_fr" "reference_guide_en" "reference_guide_fr"
#> [7] "licence_en" "licence_fr" "licence_url_en"
#> [10] "licence_url_fr" "languages" "attribution_en"
#> [13] "attribution_fr" "notes_en" "notes_fr"
#> [16] "columns" "primary_key" "geometry_repairs"
#> [19] "processing"
db_metadata$columns |> lapply(names) |> head(1)
#> $DBUID
#> [1] "label_en" "label_fr"Monolingual metadata stores plain title and
label entries instead, and the two mix: a dataset with a
language of its own, which the language entry then names,
can carry a plain title with a title_fr next
to it. Columns follow the same convention, the file describing the
census profile characteristics below holds the names as
name_en and name_fr.
data_to_parquet() refuses column labels in a language
the metadata doesn’t declare, so a bilingual dataset has to say
languages = c("en", "fr"). Only a declared suffix marks a
variant: source_url and licence_url above are
entries of their own, not a source and a
licence in a language called url, and asking
for a language the dataset doesn’t declare returns it as stored.
Language codes are lowercase, two or three letters with optional subtags
like pt-br, and the Languages section of
?data_to_parquet has the full set of rules.
The stored metadata is the full set. Passing language
resolves it to one: parquet_metadata(language = "fr") picks
the French variant of every entry and column label, drops the others and
records the language in the language entry, so the result
is a complete description of the dataset in that language. An entry
comes from the variant in the language asked for, else from the plain
entry, else from the variant in the dataset’s own language.
parquet_metadata(db_url, language = "fr")
#> <parquet metadata: 12 entries>
#> licence Licence ouverte de Statistique Canada
#> licence_url https://www.statcan.gc.ca/fr/reference/licence
#> title Recensement de 2021 – Fichiers des limites – Îlots de diffusi…
#> source_url https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-p…
#> reference_guide https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/g…
#> attribution Statistique Canada, Fichier des limites des îlots de diffusio…
#> notes Les identificateurs des géographies englobantes proviennent d…
#> columns <13 columns, see below>
#> primary_key DBUID
#> geometry_repairs {"columns":"DBUID","rows":["10010273010","10010667008","10010…
#> processing Repaired 830 of 498,547 geometries that were invalid, via sf:…
#> language fr
#>
#> columns
#> DBUID Identificateur unique de l'îlot de diffusion
#> DAUID Identificateur unique de l'aire de diffusion
#> CTUID Identificateur unique du secteur de recensement
#> CSDUID Identificateur unique de la subdivision de recensement
#> CMAUID Identificateur unique de la région métropolitaine de recensement ou…
#> CMAPUID Identificateur unique de la partie provinciale de la RMR/AR: Code d…
#> CDUID Identificateur unique de la division de recensement
#> PRUID Identificateur unique de la province ou du territoire
#> DGUID Identificateur unique de géographie de diffusion (IDUGD)
#> DBRPLAMX Point représentatif, x (Lambert)
#> DBRPLAMY Point représentatif, y (Lambert)
#> LANDAREA Superficie des terres (km²)
#> geometry Limites, projection conique conforme de Lambert de Statistique Cana…That is what collect_parquet(language = ) attaches to
the collected data, together with the labels in that language. The
columns follow the same rule: parquet_tbl(language = )
selects the translated columns of the asked language under their plain
name on the lazy table, name_en or name_fr
becomes name, and leaves the others out, before anything is
read, so queries use name and the other language never gets
downloaded. The default for the language is the
r2parquet.language option, which we set at the top of this
article: without it, and without a language entry, a
dataset like this one has no plain text to fall back on, so no labels
get attached and parquet_sources_attribution() has no
attribution to return, which a message points out once per session. Set
the option once per session or in your .Rprofile, or pass
language to parquet_tbl(),
collect_parquet(), parquet_sources() and
parquet_sources_attribution() where one dataset should come
out in another language; language = NULL gives the columns
as stored, and select_language() does the swap later on any
lazy table, joins included. parquet_metadata() ignores the
option on purpose, so the full metadata of a lazy table or path is
always one call away.
A directory with several datasets
Directories on R2 can be listed with the credentials of the bucket,
but not over https, so a public bucket needs some other way of telling a
client what a directory holds. That is the manifest, a
_manifest.json file next to the data, which
upload_parquet() and data_to_parquet() write
for the directories they upload and write_manifest() adds
to directories that were uploaded by other means. A manifest lists every
parquet file below its directory with its size, etag and modification
time, and copies the metadata from the files’ footers, so clients get
all of it with a single request.
The census/2021 directory of the public bucket holds
several datasets, single files like the boundary files and the
geographic attribute file, and partitioned datasets that are directories
of files, and one manifest listing all of them. We show it with the file
entries cut down to three, a single file and the first file of each of
the two partitioned datasets, and the list of repaired geometries cut
down to three rows.
census_url <- "https://data.mountainmath.ca/census/2021/"
manifest <- jsonlite::fromJSON(paste0(census_url, "_manifest.json"), simplifyVector = FALSE)
length(manifest$files)
#> [1] 31
paths <- sapply(manifest$files, function(f) f$path)
manifest$files <- manifest$files[match(c("ldb_000b21f_e.parquet", "lada000b21f_e/PRUID=10/data_0.parquet",
"profile_ADA/PRUID=10/data_0.parquet"), paths)]
manifest$files <- lapply(manifest$files, function(f) {
if (!is.null(f$metadata$geometry_repairs)) f$metadata$geometry_repairs$rows <- head(f$metadata$geometry_repairs$rows, 3)
f
})
jsonlite::toJSON(manifest, auto_unbox = TRUE, pretty = TRUE)
#> {
#> "directory": true,
#> "metadata": {
#> "licence_en": "Statistics Canada Open Licence",
#> "licence_fr": "Licence ouverte de Statistique Canada",
#> "licence_url_en": "https://www.statcan.gc.ca/en/terms-conditions/open-licence",
#> "licence_url_fr": "https://www.statcan.gc.ca/fr/reference/licence",
#> "languages": [
#> "en",
#> "fr"
#> ]
#> },
#> "files": [
#> {
#> "path": "ldb_000b21f_e.parquet",
#> "size": 330189472,
#> "etag": "e8379be49255c17c6bc757e1baf0f757",
#> "modified": "2026-10-24T07:27:37Z",
#> "metadata": {
#> "title_en": "2021 Census – Boundary files – Dissemination blocks",
#> "title_fr": "Recensement de 2021 – Fichiers des limites – Îlots de diffusion",
#> "source_url_en": "https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/ldb_000b21f_e.zip",
#> "source_url_fr": "https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/ldb_000b21f_f.zip",
#> "reference_guide_en": "https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/guide/index2021-eng.cfm?year=21",
#> "reference_guide_fr": "https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/guide/index2021-fra.cfm?year=21",
#> "attribution_en": "Statistics Canada, 2021 Census dissemination block boundary file ldb_000b21f_e",
#> "attribution_fr": "Statistique Canada, Fichier des limites des îlots de diffusion du Recensement de 2021, ldb_000b21f_e",
#> "notes_en": "The identifiers of the containing geographies come from the Geographic Attribute File (92-151-X).",
#> "notes_fr": "Les identificateurs des géographies englobantes proviennent du Fichier des attributs géographiques (92-151-X).",
#> "columns": {
#> "DBUID": {
#> "label_en": "Dissemination block ID",
#> "label_fr": "Identificateur unique de l'îlot de diffusion"
#> },
#> "DAUID": {
#> "label_en": "Dissemination area ID",
#> "label_fr": "Identificateur unique de l'aire de diffusion"
#> },
#> "CTUID": {
#> "label_en": "Census tract ID",
#> "label_fr": "Identificateur unique du secteur de recensement"
#> },
#> "CSDUID": {
#> "label_en": "Census subdivision ID",
#> "label_fr": "Identificateur unique de la subdivision de recensement"
#> },
#> "CMAUID": {
#> "label_en": "Census metropolitan area or census agglomeration ID",
#> "label_fr": "Identificateur unique de la région métropolitaine de recensement ou de l'agglomération de recensement"
#> },
#> "CMAPUID": {
#> "label_en": "CMA/CA provincial part ID",
#> "label_fr": "Identificateur unique de la partie provinciale de la RMR/AR",
#> "description_en": "Province code followed by the CMA/CA code, unique within CMA/CAs spanning two provinces",
#> "description_fr": "Code de la province suivi du code de la RMR/AR, distinct pour les RMR/AR chevauchant deux provinces"
#> },
#> "CDUID": {
#> "label_en": "Census division ID",
#> "label_fr": "Identificateur unique de la division de recensement"
#> },
#> "PRUID": {
#> "label_en": "Province or territory ID",
#> "label_fr": "Identificateur unique de la province ou du territoire"
#> },
#> "DGUID": {
#> "label_en": "Dissemination geography unique identifier (DGUID)",
#> "label_fr": "Identificateur unique de géographie de diffusion (IDUGD)"
#> },
#> "DBRPLAMX": {
#> "label_en": "Representative point, x (Lambert)",
#> "label_fr": "Point représentatif, x (Lambert)"
#> },
#> "DBRPLAMY": {
#> "label_en": "Representative point, y (Lambert)",
#> "label_fr": "Point représentatif, y (Lambert)"
#> },
#> "LANDAREA": {
#> "label_en": "Land area (km²)",
#> "label_fr": "Superficie des terres (km²)"
#> },
#> "geometry": {
#> "label_en": "Boundary, Statistics Canada Lambert conformal conic",
#> "label_fr": "Limites, projection conique conforme de Lambert de Statistique Canada"
#> }
#> },
#> "primary_key": [
#> "DBUID"
#> ],
#> "geometry_repairs": {
#> "columns": [
#> "DBUID"
#> ],
#> "rows": [
#> [
#> "10010273010"
#> ],
#> [
#> "10010667008"
#> ],
#> [
#> "10010817014"
#> ]
#> ]
#> },
#> "processing": "Repaired 830 of 498,547 geometries that were invalid, via sf::st_make_valid (GEOS 3.13.0), keeping multi geometry types. The repaired rows are listed in geometry_repairs by DBUID."
#> }
#> },
#> {
#> "path": "lada000b21f_e/PRUID=10/data_0.parquet",
#> "size": 11809465,
#> "etag": "db06997fb4281ca229d2d47a83dfbb3b",
#> "modified": "2026-10-24T07:29:48Z",
#> "metadata": {
#> "title_en": "2021 Census – Boundary files – Aggregate dissemination areas",
#> "title_fr": "Recensement de 2021 – Fichiers des limites – Aires de diffusion agrégées",
#> "source_url_en": "https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lada000b21f_e.zip",
#> "source_url_fr": "https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lada000b21f_f.zip",
#> "reference_guide_en": "https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/guide/index2021-eng.cfm?year=21",
#> "reference_guide_fr": "https://www12.statcan.gc.ca/census-recensement/2021/geo/ref/guide/index2021-fra.cfm?year=21",
#> "attribution_en": "Statistics Canada, 2021 Census aggregate dissemination area boundary file lada000b21f_e",
#> "attribution_fr": "Statistique Canada, Fichier des limites des aires de diffusion agrégées du Recensement de 2021, lada000b21f_e",
#> "columns": {
#> "ADAUID": {
#> "label_en": "Aggregate dissemination area ID",
#> "label_fr": "Identificateur unique de l'aire de diffusion agrégée"
#> },
#> "DGUID": {
#> "label_en": "Dissemination geography unique identifier (DGUID)",
#> "label_fr": "Identificateur unique de géographie de diffusion (IDUGD)"
#> },
#> "LANDAREA": {
#> "label_en": "Land area (km²)",
#> "label_fr": "Superficie des terres (km²)"
#> },
#> "geometry": {
#> "label_en": "Boundary, Statistics Canada Lambert conformal conic",
#> "label_fr": "Limites, projection conique conforme de Lambert de Statistique Canada"
#> },
#> "PRUID": {
#> "label_en": "Province or territory ID",
#> "label_fr": "Identificateur unique de la province ou du territoire"
#> }
#> },
#> "primary_key": [
#> "ADAUID"
#> ],
#> "geometry_repairs": {
#> "columns": [
#> "ADAUID"
#> ],
#> "rows": [
#> [
#> "24230030"
#> ],
#> [
#> "24370011"
#> ],
#> [
#> "24720013"
#> ]
#> ]
#> },
#> "processing": "Repaired 44 of 5,433 geometries that were invalid, via sf::st_make_valid (GEOS 3.13.0), keeping multi geometry types. The repaired rows are listed in geometry_repairs by ADAUID."
#> }
#> },
#> {
#> "path": "profile_ADA/PRUID=10/data_0.parquet",
#> "size": 857603,
#> "etag": "22bb1efe5c64ef56364b8f5edc1289fa",
#> "modified": "2026-10-24T07:31:04Z",
#> "metadata": {
#> "title_en": "2021 Census Profile – Aggregate dissemination areas",
#> "title_fr": "Profil du recensement de 2021 – Aires de diffusion agrégées",
#> "source_url_en": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/download-telecharger/comp/GetFile.cfm?Lang=E&FILETYPE=CSV&GEONO=012",
#> "source_url_fr": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/download-telecharger/comp/GetFile.cfm?Lang=F&FILETYPE=CSV&GEONO=012",
#> "reference_guide_en": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/about-apropos.cfm?Lang=E",
#> "reference_guide_fr": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/about-apropos.cfm?Lang=F",
#> "attribution_en": "Statistics Canada, 2021 Census Profile, product 98-401-X2021012",
#> "attribution_fr": "Statistique Canada, Profil du recensement de 2021, produit 98-401-X2021012",
#> "notes_en": "Symbols: x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised. Characteristics are described in profile_characteristics.parquet, geographies in profile_ADA_geographies.parquet.",
#> "notes_fr": "Symboles : x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé. Les caractéristiques sont décrites dans profile_characteristics.parquet, les géographies dans profile_ADA_geographies.parquet.",
#> "columns": {
#> "geo_uid": {
#> "label_en": "Aggregate dissemination area ID",
#> "label_fr": "Identificateur unique de l'aire de diffusion agrégée"
#> },
#> "PRUID": {
#> "label_en": "Province or territory ID",
#> "label_fr": "Identificateur unique de la province ou du territoire"
#> },
#> "characteristic_id": {
#> "label_en": "Characteristic ID",
#> "label_fr": "Identificateur de la caractéristique",
#> "description_en": "See profile_characteristics.parquet",
#> "description_fr": "Voir profile_characteristics.parquet"
#> },
#> "total": {
#> "label_en": "Count, total",
#> "label_fr": "Chiffre, total"
#> },
#> "men": {
#> "label_en": "Count, men+",
#> "label_fr": "Chiffre, hommes+"
#> },
#> "women": {
#> "label_en": "Count, women+",
#> "label_fr": "Chiffre, femmes+"
#> },
#> "rate_total": {
#> "label_en": "Rate, total",
#> "label_fr": "Taux, total"
#> },
#> "rate_men": {
#> "label_en": "Rate, men+",
#> "label_fr": "Taux, hommes+"
#> },
#> "rate_women": {
#> "label_en": "Rate, women+",
#> "label_fr": "Taux, femmes+"
#> },
#> "symbol_total": {
#> "label_en": "Symbol for the total count",
#> "label_fr": "Symbole du chiffre total",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_men": {
#> "label_en": "Symbol for the men+ count",
#> "label_fr": "Symbole du chiffre hommes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_women": {
#> "label_en": "Symbol for the women+ count",
#> "label_fr": "Symbole du chiffre femmes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_rate_total": {
#> "label_en": "Symbol for the total rate",
#> "label_fr": "Symbole du taux total",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_rate_men": {
#> "label_en": "Symbol for the men+ rate",
#> "label_fr": "Symbole du taux hommes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_rate_women": {
#> "label_en": "Symbol for the women+ rate",
#> "label_fr": "Symbole du taux femmes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> }
#> },
#> "primary_key": [
#> "geo_uid",
#> "characteristic_id"
#> ]
#> }
#> }
#> ]
#> }The manifest’s own metadata holds the entries all files
agree on, the licence here, which is what a client can take as the
metadata of the directory, and each file lists the entries that differ,
its title, source and attribution, and for the files that have them the
column descriptions. directory says that the manifest lists
everything below its directory, as the manifests on R2 do; the local
cache writes manifests with directory: false for
directories in which only single files were cached. The format is
documented in ?write_manifest.
list_parquet() lists a directory at a URL from its
manifest, with the files’ size and etag, here shown with the paths
relative to the directory.
list_parquet(census_url) |>
mutate(path = sub(census_url, "", path, fixed = TRUE)) |>
select(path, size, modified)
#> # A tibble: 31 × 3
#> path size modified
#> <chr> <chr> <dttm>
#> 1 2021_92-151_X.parquet 10.2 MB 2026-10-24 07:40:02
#> 2 lada000b21f_e/PRUID=10/data_0.parquet 11.8 MB 2026-10-24 07:29:48
#> 3 lada000b21f_e/PRUID=11/data_0.parquet 451.5 kB 2026-10-24 07:29:48
#> 4 lada000b21f_e/PRUID=12/data_0.parquet 3.4 MB 2026-10-24 07:29:48
#> 5 lada000b21f_e/PRUID=13/data_0.parquet 1.8 MB 2026-10-24 07:29:48
#> 6 lada000b21f_e/PRUID=24/data_0.parquet 25.3 MB 2026-10-24 07:29:48
#> 7 lada000b21f_e/PRUID=35/data_0.parquet 40.4 MB 2026-10-24 07:29:48
#> 8 lada000b21f_e/PRUID=46/data_0.parquet 2.9 MB 2026-10-24 07:29:48
#> 9 lada000b21f_e/PRUID=47/data_0.parquet 1.2 MB 2026-10-24 07:29:48
#> 10 lada000b21f_e/PRUID=48/data_0.parquet 2.3 MB 2026-10-24 07:29:48
#> # ℹ 21 more rowsparquet_metadata() reads the manifest for paths on R2
and at URLs, instead of the footers: for a file it looks for the
manifest of the file’s directory, or of the nearest directory above it
that has one, and takes the file’s entry, a single request that is the
same for every file of a dataset. The result is the same as reading the
footer above, as long as the manifest is current.
parquet_metadata(db_url)[c("title_en", "title_fr", "primary_key", "processing")]
#> <parquet metadata: 4 entries>
#> title_en 2021 Census – Boundary files – Dissemination blocks
#> title_fr Recensement de 2021 – Fichiers des limites – Îlots de diffusion
#> primary_key DBUID
#> processing Repaired 830 of 498,547 geometries that were invalid, via sf::st_m…The manifest is current as long as the files only change through
upload_parquet(), data_to_parquet() and
remove_parquet(), which keep the manifests of the
directories they touch and of the directories above up to date. Files
changed with other tools are not reflected until
write_manifest() rebuilds the manifest from a listing;
from_files = TRUE bypasses the manifest and reads the
footers.
Manifests are per directory and cover everything below, so a manifest
in a higher level directory covers all the datasets below it, as
census/2021 does here, and
write_manifest("r2://bucket") covers the whole bucket.
Every upload and removal below a manifest patches it, so the higher the
manifest, the more changes it sees.
A partitioned dataset
A dataset can also be a directory of files, hive partitioned by the
values of one or more columns, which is how
data_to_parquet(partition_by = ) writes it. The 2021 census
profile for aggregate dissemination areas in the public bucket is such a
dataset: one row per area and characteristic, partitioned by province,
with the characteristics described in a small file next to it.
Each partition is a directory named after the column and its value,
holding the parquet file for that value. The partition column is not
stored in the files, readers recover it from the directory names, as
strings, so a province code like "10" stays a string and
keeps its leading zeros. The dataset’s manifest lists the partition
files.
profile_url <- "https://data.mountainmath.ca/census/2021/profile_ADA/"
list_parquet(profile_url) |>
mutate(path = sub(profile_url, "", path, fixed = TRUE)) |>
select(path, size, modified)
#> # A tibble: 13 × 3
#> path size modified
#> <chr> <chr> <dttm>
#> 1 PRUID=10/data_0.parquet 857.6 kB 2026-10-24 07:31:04
#> 2 PRUID=11/data_0.parquet 298.8 kB 2026-10-24 07:31:04
#> 3 PRUID=12/data_0.parquet 1.5 MB 2026-10-24 07:31:04
#> 4 PRUID=13/data_0.parquet 1.3 MB 2026-10-24 07:31:04
#> 5 PRUID=24/data_0.parquet 14.8 MB 2026-10-24 07:31:04
#> 6 PRUID=35/data_0.parquet 23 MB 2026-10-24 07:31:04
#> 7 PRUID=46/data_0.parquet 2.4 MB 2026-10-24 07:31:04
#> 8 PRUID=47/data_0.parquet 2.7 MB 2026-10-24 07:31:04
#> 9 PRUID=48/data_0.parquet 5.8 MB 2026-10-24 07:31:04
#> 10 PRUID=59/data_0.parquet 9.8 MB 2026-10-24 07:31:04
#> 11 PRUID=60/data_0.parquet 244.9 kB 2026-10-24 07:31:04
#> 12 PRUID=61/data_0.parquet 307 kB 2026-10-24 07:31:04
#> 13 PRUID=62/data_0.parquet 270.5 kB 2026-10-24 07:31:04The files were written together, so they share their metadata,
columns included, and the manifest carries it once, at the
dataset level, while the file entries only list the files;
columns is never repeated per file. We show the first two
entries.
manifest <- jsonlite::fromJSON(paste0(profile_url, "_manifest.json"), simplifyVector = FALSE)
manifest$files <- manifest$files[1:2]
jsonlite::toJSON(manifest, auto_unbox = TRUE, pretty = TRUE)
#> {
#> "directory": true,
#> "metadata": {
#> "title_en": "2021 Census Profile – Aggregate dissemination areas",
#> "title_fr": "Profil du recensement de 2021 – Aires de diffusion agrégées",
#> "source_url_en": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/download-telecharger/comp/GetFile.cfm?Lang=E&FILETYPE=CSV&GEONO=012",
#> "source_url_fr": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/download-telecharger/comp/GetFile.cfm?Lang=F&FILETYPE=CSV&GEONO=012",
#> "reference_guide_en": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/about-apropos.cfm?Lang=E",
#> "reference_guide_fr": "https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/about-apropos.cfm?Lang=F",
#> "licence_en": "Statistics Canada Open Licence",
#> "licence_fr": "Licence ouverte de Statistique Canada",
#> "licence_url_en": "https://www.statcan.gc.ca/en/terms-conditions/open-licence",
#> "licence_url_fr": "https://www.statcan.gc.ca/fr/reference/licence",
#> "languages": [
#> "en",
#> "fr"
#> ],
#> "attribution_en": "Statistics Canada, 2021 Census Profile, product 98-401-X2021012",
#> "attribution_fr": "Statistique Canada, Profil du recensement de 2021, produit 98-401-X2021012",
#> "notes_en": "Symbols: x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised. Characteristics are described in profile_characteristics.parquet, geographies in profile_ADA_geographies.parquet.",
#> "notes_fr": "Symboles : x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé. Les caractéristiques sont décrites dans profile_characteristics.parquet, les géographies dans profile_ADA_geographies.parquet.",
#> "columns": {
#> "geo_uid": {
#> "label_en": "Aggregate dissemination area ID",
#> "label_fr": "Identificateur unique de l'aire de diffusion agrégée"
#> },
#> "PRUID": {
#> "label_en": "Province or territory ID",
#> "label_fr": "Identificateur unique de la province ou du territoire"
#> },
#> "characteristic_id": {
#> "label_en": "Characteristic ID",
#> "label_fr": "Identificateur de la caractéristique",
#> "description_en": "See profile_characteristics.parquet",
#> "description_fr": "Voir profile_characteristics.parquet"
#> },
#> "total": {
#> "label_en": "Count, total",
#> "label_fr": "Chiffre, total"
#> },
#> "men": {
#> "label_en": "Count, men+",
#> "label_fr": "Chiffre, hommes+"
#> },
#> "women": {
#> "label_en": "Count, women+",
#> "label_fr": "Chiffre, femmes+"
#> },
#> "rate_total": {
#> "label_en": "Rate, total",
#> "label_fr": "Taux, total"
#> },
#> "rate_men": {
#> "label_en": "Rate, men+",
#> "label_fr": "Taux, hommes+"
#> },
#> "rate_women": {
#> "label_en": "Rate, women+",
#> "label_fr": "Taux, femmes+"
#> },
#> "symbol_total": {
#> "label_en": "Symbol for the total count",
#> "label_fr": "Symbole du chiffre total",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_men": {
#> "label_en": "Symbol for the men+ count",
#> "label_fr": "Symbole du chiffre hommes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_women": {
#> "label_en": "Symbol for the women+ count",
#> "label_fr": "Symbole du chiffre femmes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_rate_total": {
#> "label_en": "Symbol for the total rate",
#> "label_fr": "Symbole du taux total",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_rate_men": {
#> "label_en": "Symbol for the men+ rate",
#> "label_fr": "Symbole du taux hommes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> },
#> "symbol_rate_women": {
#> "label_en": "Symbol for the women+ rate",
#> "label_fr": "Symbole du taux femmes+",
#> "description_en": "x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised",
#> "description_fr": "x confidentiel en vertu des dispositions de la Loi sur la statistique, F trop peu fiable pour être publié, .. indisponible pour une période de référence précise, ... n'ayant pas lieu de figurer, r révisé"
#> }
#> },
#> "primary_key": [
#> "geo_uid",
#> "characteristic_id"
#> ]
#> },
#> "files": [
#> {
#> "path": "PRUID=10/data_0.parquet",
#> "size": 857603,
#> "etag": "22bb1efe5c64ef56364b8f5edc1289fa",
#> "modified": "2026-10-24T07:31:04Z"
#> },
#> {
#> "path": "PRUID=11/data_0.parquet",
#> "size": 298761,
#> "etag": "12d0126f6aea0d96e14fb763af501008",
#> "modified": "2026-10-24T07:31:04Z"
#> }
#> ]
#> }parquet_metadata() on the directory returns that shared
metadata, from the manifest. The columns entry prints as a
table of the columns with their labels and descriptions.
parquet_metadata(profile_url)
#> <parquet metadata: 17 entries>
#> en fr
#> title 2021 Census Profile – Aggrega… Profil du recensement de 2021…
#> source_url https://www12.statcan.gc.ca/c… https://www12.statcan.gc.ca/c…
#> reference_guide https://www12.statcan.gc.ca/c… https://www12.statcan.gc.ca/c…
#> licence Statistics Canada Open Licence Licence ouverte de Statistiqu…
#> licence_url https://www.statcan.gc.ca/en/… https://www.statcan.gc.ca/fr/…
#> languages en, fr
#> attribution Statistics Canada, 2021 Censu… Statistique Canada, Profil du…
#> notes Symbols: x suppressed to meet… Symboles : x confidentiel en …
#> columns <15 columns, see below>
#> primary_key geo_uid, characteristic_id
#>
#> columns en fr
#> geo_uid Aggregate dissemination are… Identificateur unique de l'…
#> PRUID Province or territory ID Identificateur unique de la…
#> characteristic_id Characteristic ID: See prof… Identificateur de la caract…
#> total Count, total Chiffre, total
#> men Count, men+ Chiffre, hommes+
#> women Count, women+ Chiffre, femmes+
#> rate_total Rate, total Taux, total
#> rate_men Rate, men+ Taux, hommes+
#> rate_women Rate, women+ Taux, femmes+
#> symbol_total Symbol for the total count:… Symbole du chiffre total: x…
#> symbol_men Symbol for the men+ count: … Symbole du chiffre hommes+:…
#> symbol_women Symbol for the women+ count… Symbole du chiffre femmes+:…
#> symbol_rate_total Symbol for the total rate: … Symbole du taux total: x co…
#> symbol_rate_men Symbol for the men+ rate: x… Symbole du taux hommes+: x …
#> symbol_rate_women Symbol for the women+ rate:… Symbole du taux femmes+: x …parquet_tbl() reads the directory as one table, with
PRUID coming from the paths, and a filter on it only opens
the files of the matching partitions. Within a partition the rows are
sorted by characteristic_id, so a filter on it only
downloads the row groups holding that characteristic. The literal is
written as 1L, an integer like the column: a plain
1 is a double in R, which would make DuckDB cast the column
instead of using its statistics and download every row group, so the
package renders whole numbers in comparisons as integers, with a message
the first time, see ?parquet_tbl. Partition columns are
strings, so PRUID is compared with "59"; a
number there would make DuckDB cast the column instead and read every
partition, which is why the package renders numbers against partition
columns as strings as well.
parquet_tbl(profile_url) |>
filter(PRUID == "59", characteristic_id == 1L) |>
summarise(Population = sum(total, na.rm = TRUE), Areas = n(), .by = PRUID) |>
collect()
#> # A tibble: 1 × 3
#> PRUID Population Areas
#> <chr> <dbl> <dbl>
#> 1 59 5000879 1008collect_parquet() attaches the metadata to the result
and the column labels to the columns, and
parquet_metadata() on a lazy table reads the manifest of
its source. Here we join the characteristic names from the file next to
the dataset, both tables sit on the same DuckDB connection. The
characteristics file stores the names in both languages,
name_en and name_fr, and the session language
makes parquet_tbl() select name_en as
name, so the query selects name like any other
column and name_fr never gets read.
characteristics_url <- "https://data.mountainmath.ca/census/2021/profile_characteristics.parquet"
characteristics <- parquet_tbl(characteristics_url)
colnames(characteristics)
#> [1] "characteristic_id" "name" "level"
#> [4] "note"
colnames(parquet_tbl(characteristics_url, language = NULL))
#> [1] "characteristic_id" "name_en" "name_fr"
#> [4] "level" "note"
area_query <- parquet_tbl(profile_url) |>
filter(PRUID == "59", geo_uid == "59150002", characteristic_id <= 8L) |>
inner_join(characteristics, by = "characteristic_id") |>
select(geo_uid, characteristic_id, name, total)
area <- collect_parquet(area_query)
area
#> # A tibble: 8 × 4
#> geo_uid characteristic_id name total
#> <chr> <int> <chr> <dbl>
#> 1 59150002 1 Population, 2021 4256
#> 2 59150002 2 Population, 2016 3680
#> 3 59150002 3 Population percentage change, 2016 to 2021 15.7
#> 4 59150002 4 Total private dwellings 2036
#> 5 59150002 5 Private dwellings occupied by usual residen… 1724
#> 6 59150002 6 Population density per square kilometre 84.9
#> 7 59150002 7 Land area in square kilometres 50.1
#> 8 59150002 8 Total - Age groups of the population - 100%… 4260
attr(area$total, "label")
#> [1] "Count, total"
parquet_metadata(area)$notes
#> [1] "Symbols: x suppressed to meet the confidentiality requirements of the Statistics Act, F too unreliable to be published, .. not available for a specific reference period, ... not applicable, r revised. Characteristics are described in profile_characteristics.parquet, geographies in profile_ADA_geographies.parquet."The joined table contributes its own metadata,
parquet_sources() lists every table that went into the
result, with its attribution, and
parquet_sources_attribution() joins them for a figure
caption. Reading the tables in French gives the French names, labels and
attributions with the same query; select_language() does
the same swap on a table read with language = NULL, on
joins too, for tables that were created before the language was
decided.
parquet_sources_attribution(area)
#> [1] "Statistics Canada, 2021 Census Profile, product 98-401-X2021012"
area_fr <- parquet_tbl(profile_url, language = "fr") |>
filter(PRUID == "59", geo_uid == "59150002", characteristic_id <= 8L) |>
inner_join(parquet_tbl(characteristics_url, language = "fr"), by = "characteristic_id") |>
select(geo_uid, characteristic_id, name, total) |>
collect_parquet()
area_fr
#> # A tibble: 8 × 4
#> geo_uid characteristic_id name total
#> <chr> <int> <chr> <dbl>
#> 1 59150002 1 Population, 2021 4256
#> 2 59150002 2 Population, 2016 3680
#> 3 59150002 3 Variation en pourcentage de la population, … 15.7
#> 4 59150002 4 Total des logements privés 2036
#> 5 59150002 5 Logements privés occupés par des résidents … 1724
#> 6 59150002 6 Densité de la population au kilomètre carré 84.9
#> 7 59150002 7 Superficie des terres en kilomètres carrés 50.1
#> 8 59150002 8 Total - Groupes d'âge de la population - Do… 4260
attr(area_fr$total, "label")
#> [1] "Chiffre, total"
parquet_sources_attribution(area_fr)
#> [1] "Statistique Canada, Profil du recensement de 2021, produit 98-401-X2021012"The metadata attached to area_fr is the French
description, with language set to fr, since
the columns were selected in French; writing it out with
data_to_parquet(area_fr, metadata = parquet_metadata(area_fr))
gives a French dataset. The full bilingual metadata stays available from
the lazy table, parquet_metadata(area_query).
Writing a partitioned dataset to R2 with
data_to_parquet(partition_by = ) writes the files and the
manifest in one step, and removes the partitions of an earlier write
that the new one no longer produces, so the manifest never lists
partitions that are gone. Uploading a single partition into the
directory later, with data_to_parquet() or
upload_parquet(), patches the manifest with the new file
instead of rebuilding it, and so does every upload and removal for the
manifests of the directories above, which is how the
census/2021 manifest keeps listing the partition files.
The local cache
cache_parquet() and
parquet_tbl(cache = TRUE) mirror remote files and
directories into a local cache directory, in the same layout as on R2,
and keep local manifests of the same format next to the copies, with the
remote etags they were downloaded with. That is how the package knows
whether a copy is current without comparing files, and how
list_parquet_cache() knows what the cache holds. The Caching article covers the cache.