Skip to contents

r2parquet

Publish parquet and GeoParquet files on Cloudflare R2 and query them lazily from R. Amazon S3 and other S3 compatible services like MinIO work too, see Amazon S3 and other S3 compatible services.

Reading public Parquet and GeoParquet data from R2

  • Reading public URLs and local files: parquet_tbl(), no credentials needed. Returns a lazy DuckDB table, only the parts of the files a query needs get downloaded.
  • Spatial queries: sf::st_filter() on any lazy DuckDB table, skipping row groups outside the query, collect_parquet() collects as sf if the table has geometry.
  • Metadata: parquet_metadata() reads source, license, notes and column labels from the parquet footer, collect_parquet() attaches them to the collected data, where parquet_metadata() reads them back too. Datasets can carry labels and text columns in several languages, parquet_tbl(language = "fr") (or the r2parquet.language option, or select_language() on a lazy table) selects the columns of one language before anything is read and collect_parquet() attaches its labels.
  • Working locally: parquet_tbl(cache = TRUE) downloads the files once into a local cache directory and keeps them current via their etags, for data queried regularly. Once cached, parquet_tbl() reads the local copy by default as long as it is current (checked once per session, clear_cache_checks() forgets the checks), warns when it is out of date, and reads it without checking with cache = "offline". cache_parquet() syncs the cache on its own, list_parquet_cache() and remove_parquet_from_cache() show and prune it, and list_parquet() tells whether the cached copies are current. See Caching.
  • Joining with other data: all tables share one DuckDB connection, parquet_connection(), so they can be joined with each other and with data frames copied there via dplyr::copy_to(). parquet_tbl(con = ) puts a table on a connection of your own instead.

Uploading and managing data on R2

  • Buckets: create_r2_bucket() creates private or public buckets, public via the r2.dev URL or a custom domain.
  • Files: upload_parquet() uploads single parquet files or directories of (partitioned) parquet files, with remove_stale = TRUE removing the files on R2 that are no longer in the directory, single files of any other type, like a JSON index, go up with the content type for their extension, remove_parquet() removes parquet files and directories (and single files of any type) and list_parquet() lists parquet files with size, modification time, etag and cache status. Uploads and removals purge the changed files from the Cloudflare cache of the bucket’s custom domains. Directory uploads write a _manifest.json listing the files with their footer metadata, which lets public readers cache the directory and see what a dataset holds without opening the files, write_manifest() adds it to existing directories.
  • Writing: data_to_parquet() writes data frames, sf objects and lazy DuckDB tables to parquet, optionally sorted (sort_by) and partitioned (partition_by), so remote queries only download the row groups they need. sf objects become GeoParquet 2.0 files sorted along a Hilbert curve. Given an r2://bucket/path destination it writes and uploads in one go, replacing earlier uploads including stale partitions. metadata like the source URL, license, column labels and the primary key goes into the parquet footer. Invalid geometries get repaired, and the repairs recorded in the metadata.
  • Reading private buckets: parquet_tbl("r2://bucket/path").
  • Sharing private files: presigned_url() creates time limited download (or upload) URLs for files in private buckets.
  • Amazon S3 and other S3 compatible services: all of the above works on s3://bucket/path as well, with the usual AWS environment variables. set_s3_public_access() makes a prefix of an S3 bucket public, so public and private data can share a bucket. See Amazon S3 and other S3 compatible services.

Installation 

remotes::install_github("mountainMath/r2parquet")

All of R2 is reached through DuckDB and signed https requests, there are no optional dependencies.

Credentials

Set these environment variables, e.g. in ~/.Renviron (usethis::edit_r_environ() opens it, restart R afterwards):

Variable Needed for
R2_ACCOUNT_ID everything on R2
R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY reading, uploading and removing files (S3 API credentials of an R2 API token)
R2_API_TOKEN create_r2_bucket() (Admin Read & Write); Zone Read for custom domains; Zone Cache Purge for cache purging on upload/removal
R2PARQUET_PURGE_ORIGINS optional, comma separated origins of web pages reading the bucket from a browser, whose cached copies get purged as well; not needed for reading with this package, DuckDB or other clients that send no Origin header

See Setting up R2 for the token permissions, caching and rate limiting of public buckets on a custom domain. s3:// paths use the standard AWS variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, and AWS_ENDPOINT_URL for services other than Amazon S3), see Amazon S3 and other S3 compatible services.

Documentation

Function reference and articles are on the package website: https://mountainmath.github.io/r2parquet/. See Accessing public buckets for an example of querying a large public GeoParquet file, and Metadata and manifests for how datasets are laid out and described.

Example

Publishing data

Needs the R2 credentials.

library(r2parquet)

create_r2_bucket("census-data", custom_domain = "data.example.com")

nc <- sf::read_sf(system.file("shape/nc.shp", package = "sf"))
data_to_parquet(nc, "r2://census-data/nc/counties.parquet")

# private access with credentials
counties <- parquet_tbl("r2://census-data/nc/counties.parquet")

# one day download link to share a file in a private bucket
presigned_url("r2://census-data/nc/counties.parquet", expires = 24 * 3600)

Accessing published data

No credentials needed, only the public URL of the file or a presigned URL shared by the publisher.

library(r2parquet)

# public access via the custom domain
counties <- parquet_tbl("https://data.example.com/nc/counties.parquet")

# or via a presigned URL
counties <- parquet_tbl("https://<account id>.r2.cloudflarestorage.com/census-data/nc/counties.parquet?X-Amz-...")

nc <- sf::read_sf(system.file("shape/nc.shp", package = "sf"))
wake <- nc[nc$NAME == "Wake", ]
counties |>
  sf::st_filter(wake) |>
  dplyr::select(NAME, BIR74, geometry) |>
  collect_parquet()