Skip to contents

list parquet files

Source: R/parquet.R
list_parquet.Rd

Lists the parquet files at paths on R2 or S3, at URLs or at local paths, or at the paths a lazy table from parquet_tbl() was created from (the remote paths for a table reading copies from the cache), with their size, modification time and etag, and whether the local cache holds a current copy. Directories get listed recursively, a path that does not exist lists nothing.

Usage

list_parquet(x)

# S3 method for class 'character'
list_parquet(x)

# S3 method for class 'tbl_lazy'
list_parquet(x)

# Default S3 method
list_parquet(x)

Arguments

x

what to list:

  • r2://bucket/path, s3://bucket/path, URL or local path of a parquet file or of a directory of (partitioned) parquet files, or a vector of these;

  • a lazy DuckDB table from parquet_tbl().

Value

a tibble with one row per parquet file and the columns

  • file: the file name;

  • size: the size formatted for reading, like 2.5 kB;

  • sizeRaw: the size in bytes;

  • modified: the modification time in UTC;

  • cache: the status of the copy in the local cache directory of cache_parquet(): "not cached", "current" when the copy has the etag of the file, "stale" when it does not, and NA for local files and where the etag is unknown;

  • etag: the etag of the file, the MD5 hash of its content for files uploaded by this package, NA for local files;

  • path: the r2://bucket/path, s3://bucket/path, URL or local path.

Details

Files at URLs, including presigned URLs, get their size and etag from a request for their first byte, directories at URLs from the manifest next to the files (see upload_parquet()). A URL with neither is listed as it is with NAs, a server that can't be reached fails with an r2parquet_network_error.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.