Skip to contents

Changelog

Source: NEWS.md

r2parquet 0.2.0

First release, moved from mountainmathHelpers.

  • Reading: parquet_tbl() gives lazy DuckDB tables on public URLs, presigned URLs and local files without credentials, and on private R2 buckets as r2://bucket/path. Directories at URLs, which can’t be listed over https, are read through the _manifest.json that uploads write next to the data, so partitioned datasets on a public bucket work like local ones. The tables live on one connection shared for the session, parquet_connection(), so they can be joined with each other and with data copied there, or on a connection of your own passed as con. Hive partition values are read as strings, whatever they look like, so PRUID=10 and PRUID=01 give the same column type, and numbers compared with a partition column for equality are rendered as strings, with a message once per session, so filter(PRUID == 59) reads that partition alone instead of casting the column and reading all of them; order comparisons need the cast spelled out, as.integer(PRUID) > 10. Whole numbers in comparisons are rendered as integers, with a message once per session, since filter(year == 2021) would otherwise compare as doubles and read every row group of an integer column instead of the matching ones; 2021L avoids the message. The arrow and polars readers of mountainmathHelpers were not carried over, neither handles GeoParquet 2.0 geometry, and uploads and listings use signed S3 requests instead of arrow, so the package has no optional dependencies.
  • Spatial queries: sf::st_filter() works on any lazy DuckDB table and only downloads the row groups that intersect the query, with the sf predicates translated to DuckDB’s spatial functions and the query transformed to the crs of the data; DuckDB’s geometry is planar, so it warns for longitude/latitude data while sf::sf_use_s2() is on, collect_parquet() collects as sf if the table has geometry. Collecting a remote table with every column it was read with gives a message, once per session, with the select() of those columns to paste in and prune, since parquet only downloads the selected columns.
  • Writing: data_to_parquet() writes data frames, sf objects and lazy DuckDB tables to parquet, optionally sorted (sort_by) and hive partitioned (partition_by). sf objects and tables with GEOMETRY columns become GeoParquet 2.0 files sorted along a Hilbert curve. Given an r2://bucket/path destination it writes and uploads in one go, removing earlier uploads at the same path so no stale partitions remain. It returns the path, ready for parquet_tbl(), with the files written in the files attribute. remove_stale = FALSE keeps partitions of earlier writes, and compression picks the codec, zstd by default.
  • Languages: datasets can carry labels and text columns in several languages. Labels and descriptions per language go into suffixed entries like label_en and label_fr, with or without a plain label, a column holding another column’s text in a language is named with the same suffix, name_fr next to name or name_en and name_fr on their own, and the language metadata entry names the language of whatever plain labels and columns there are. parquet_tbl(language = "fr"), or the r2parquet.language option, selects the columns of that language in place of the plain ones on the lazy table, leaving the other languages’ columns out of the query, so name_fr is queried as name; select_language() does the same on any lazy table from parquet_tbl() or a join of such tables, e.g. after reading with language = NULL, which gives the columns as stored. collect_parquet(language = "fr") attaches the labels of that language, falling back to the plain labels and then to the dataset language’s, labels a selected column with the label of the column it came from, describes a table whose columns were selected in a language in that language by default, and still swaps the columns at collect time where the table was read without a language. No language is privileged: a bilingual dataset can carry only suffixed labels and columns. The metadata attached to data collected in a language describes the result in that language: title_fr and attribution_fr take the place of title and attribution, the other languages’ entries are left out and the language entry names the language, so writing the result with its metadata gives a correct monolingual dataset; parquet_metadata(language =) returns the same without collecting and the metadata as stored by default, parquet_sources() and parquet_sources_attribution() take a language too (defaulting to the option, like collect_parquet()). Metadata in several languages prints consolidated, one row per entry or column with a column per language. Collecting, or asking for the sources or attribution, without a language from a dataset that has text in several languages and no plain text gives a message, once per session, saying how to set one.
  • Metadata: data_to_parquet(metadata = list(source = ..., license = ..., columns = ...)) writes source, license, notes and column labels into the parquet footer, parquet_metadata() reads them back from paths, lazy tables or collected data, for paths on R2 and at URLs from the manifest next to the data (one request for a whole dataset) unless from_files = TRUE asks for the footers themselves, and collect_parquet() attaches them to the collected data as attributes, column labels as label attributes.
  • Local cache: parquet_tbl(cache = TRUE) reads local copies of remote files for data that gets queried regularly, from the directory in R2PARQUET_CACHE_PATH or the user cache directory. Once cached, parquet_tbl() reads the copy by default as long as it is current, checking each copy once per session (clear_cache_checks() forgets the checks), and cache = "offline" reads it without checking. cache_parquet() does the syncing: it mirrors r2:// paths and URLs into the cache, compares etags so only new and changed files get downloaded, removes files that are gone remotely, checks downloads against their MD5 etag, and falls back to the cached copy with a warning when the remote can’t be reached. list_parquet_cache() lists the copies in the cache by the remote paths they stand for, remove_parquet_from_cache() removes them, and list_parquet() reports in its cache column whether a file has a current copy in the cache, a stale one or none.
  • Manifests: upload_parquet() and partitioned data_to_parquet() writes to R2 put a _manifest.json next to the data listing the parquet files with size, etag and modification time, so that directories can be cached from public URLs, where listing is not possible, and the metadata from the files’ footers, shared entries at the dataset level and the rest per file, so clients can read descriptions and column labels without opening the files (the format is documented in ?write_manifest). write_manifest() adds manifests to existing directories, reading the footers on R2, and rebuilds manifests that fell behind, remove_parquet() removes them with directories, and uploads and removals keep the manifests of the directories above (and, for directory uploads, below) the changed files current by patching them, so adding or removing a file or partition of a dataset updates the dataset’s manifest without listing it. Manifests are written conditionally, so concurrent uploads can’t overwrite each other’s updates, and a manifest that can’t be updated is reported with a warning once the data is up. Directories can’t be uploaded to the root of a bucket, write_manifest("r2://bucket") writes a manifest of the whole bucket on request.
  • Managing R2: create_r2_bucket() creates private or public buckets, upload_parquet() and remove_parquet() upload and delete files and purge them from the Cloudflare cache of the bucket’s custom domains. upload_parquet(remove_stale = TRUE) removes the parquet files below the destination that are no longer in the uploaded directory, so a rebuilt dataset gets mirrored (off by default, unlike for data_to_parquet(), which only removes its own partitions). Single files of any type that go with a dataset, like a JSON index, go up with the content type for their extension, and remove_parquet() removes them by path. File names with *, ? or [ are refused, DuckDB would read them as patterns. remove_parquet() takes several paths at once and resolves all of them before removing anything. list_parquet() lists the parquet files at R2 paths, URLs (files via their headers, directories via their manifest) or local paths, or the files a lazy table reads, with file name, size, modification time, cache status, etag and path.
  • Sharing private files: presigned_url() creates time limited download (or upload) URLs for one or more files.
  • Files on R2 are given as r2://bucket/path throughout, for reading, writing, uploading, removing and presigning.