Skip to contents

remove parquet files from Cloudflare R2, S3 or local directories

Source: R/r2.R
remove_parquet.Rd

Removes files, or all parquet files in directories of (partitioned) parquet files together with the manifests below them, from buckets or local directories, and updates the manifests of the bucket directories above. Removed files can't be recovered.

Usage

remove_parquet(path, purge_cache = TRUE)

Arguments

path

what to remove:

  • r2://bucket/path, s3://bucket/path or local path of a file or of a directory of (partitioned) parquet files, or a vector of these;

  • a lazy DuckDB table from parquet_tbl().

purge_cache

TRUE (default) or FALSE: purge the removed files from the Cloudflare cache of the bucket's custom domains. Ignored for S3 and local paths.

Value

(invisibly) the removed files as r2://bucket/path, s3://bucket/path or local paths.

Details

  • The path of a file in a bucket removes that file whatever its type, like a JSON index uploaded with upload_parquet(), or a manifest, which takes the directory's manifest away until write_manifest() or the next upload writes it again. Local files have to be parquet files, unlink() removes others.

  • The path of a directory removes its parquet files and the manifests (_manifest.json, see write_manifest()) below it; other files in it are left in place. Local directories left empty get removed as well.

  • A lazy table from parquet_tbl() removes the paths it was created from, the remote ones also when the table was read from the local cache.

All paths get resolved to files first, so a path that does not exist stops the call before anything is removed. The files are then removed in one batch per bucket. If an R2 bucket is served via a custom domain, the removed files get purged from the Cloudflare cache, with the same requirements as for upload_parquet().

Since a manifest lists all parquet files below its directory, the existing manifests of the bucket directories above removed files get updated, so removing a file or a partition from a dataset keeps the dataset's manifest current: the removed files are dropped from what the manifests record, without listing their directories, and a manifest left without files is removed. The manifests are written conditionally, like upload_parquet() writes them, and a manifest that could not be updated is reported with a warning after the files are gone. data_to_parquet() writes no manifests for local paths, so there are none to update above local files. Copies in the cache directory are removed with remove_parquet_from_cache() instead, which keeps the manifests there in line, and files at URLs can't be removed, only through their bucket path.

Credentials

Credentials come from environment variables, e.g. set in .Renviron, read at call time. Public URLs and local paths need none.

Paths on R2 (r2://bucket/path):

  • R2_ACCOUNT_ID: the R2 account id.

  • R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY: the access key id and secret access key of an R2 API token with access to the bucket.

  • R2_API_TOKEN, optional: the value of an API token with Zone Cache Purge permission (see create_r2_bucket()). Files that change on R2 then get purged from the Cloudflare cache of the bucket's custom domains, so that the new versions are served right away; the purge is skipped without it.

  • R2PARQUET_PURGE_ORIGINS, optional: the origins of web pages reading the bucket from a browser, comma separated, like https://example.com,https://www.example.com. Cloudflare caches the responses to their requests separately per Origin, which a purge of the plain URL misses, so each changed file also gets purged once per origin; copies cached for origins not listed are refreshed when the edge TTL runs out. This only concerns web apps, whose browsers send the Origin header; requests without one, like those of this package, DuckDB or curl, get the plain URL's copy, which is always purged.

Paths on S3 (s3://bucket/path), on Amazon S3 or another S3 compatible service like MinIO, use the variables AWS tools use:

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials.

  • AWS_REGION or AWS_DEFAULT_REGION: the region, default us-east-1.

  • AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL: the endpoint of other S3 compatible services, like http://localhost:9000, which get addressed with path style URLs (endpoint/bucket/key).

  • R2PARQUET_S3_URL_STYLE: path or vhost, overrides the URL style.

AWS profiles and single sign-on are not read, aws configure export-credentials --format env prints the variables for them.