Skip to contents

Download a file, through a browser if the server insists on one

Source: R/fetch.R
fetch.Rd

Fetches url to destfile. A server that serves the file to a plain request is fetched with curl and no browser is involved. A server that refuses a plain request in a way a browser gets past – a browser check (see is_challenge()) or a block page (see is_block()) – gets one: a Chrome with a throwaway profile is opened on the host, the check runs if there is one, and the file is then downloaded, through curl carrying the clearance the browser was given where the server accepts that, and in the browser itself where it does not. A clearance that curl can use is kept for the rest of the R session, so later files from the same host need no browser.

Usage

fetch(
  url,
  destfile,
  quiet = FALSE,
  timeout = 3600,
  session = NULL,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

destfile

Where to write it. A failed transfer leaves nothing there.

quiet

Suppress progress messages.

timeout

Seconds a transfer in the browser may take.

session

A browser session from browser_session(). When given, the file is downloaded in that browser without any probing, and the browser stays open afterwards; headed and useragent are then the session's. A browser that has gone away – its window closed – is opened again first.

headed

How to run the browser, when one is needed: "auto", the default, tries headless Chrome first and opens a window only when the headless run does not get through, so that a window is shown only when someone may need to click; TRUE opens a window at once and FALSE never does. Without a display "auto" is headless. A host may run a second check at the file's URL, after the one at its root; headless Chrome has passed such a check with nothing clicked, but only after about two minutes, where a window shows a box that gets it through as soon as it is clicked. "auto" waits those minutes and opens a window if they pass without; TRUE skips the wait.

useragent

The user-agent string the browser announces. Headless Chrome's own calls it HeadlessChrome, and some hosts refuse it on that word alone. TRUE, the default, drops the word, so that the string says the Chrome version and platform it is and nothing it is not; NULL leaves Chrome's own string; a string is used as given. Set before the browser's first request, and what clearance() then reports.

Value

destfile, invisibly. Errors are classed under safeconduct_error: safeconduct_unavailable when no browser can be opened here (its message says what is missing), safeconduct_not_passed when the check was not passed in time or the server blocks the browser as well, and safeconduct_download_error when the transfer itself failed.

Details

Nothing is simulated. When the check shows a "Verify you are human" box, the window stays on screen for up to two minutes for the person at the screen to click it; nothing in this package clicks it for them.

A file downloaded in the browser lands under tempdir() and is moved to destfile when complete; nothing is written next to destfile meanwhile.

Requirements

A Chrome or Chromium installation, found by chromote::find_chrome() (set the CHROMOTE_CHROME environment variable if it is somewhere unusual). By default Chrome runs headless first, and a window is opened only when the headless run ends on a check or a block and there is a display to show it on; the headed argument changes that. Headless Chrome under its own name does not pass a Cloudflare managed challenge, as of October 2026, but with the word Headless dropped from its user-agent string, which is the default, it has. Set options(safeconduct.browser = FALSE) to never open a browser. The window is 1000 by 800 pixels; options(safeconduct.window = c(width, height)) opens one of another size in pixels, and "maximized" a maximized one. Experimentally, options(safeconduct.focus = "<css selector>") pins the element that holds the check to the window's top left corner, enlarged by options(safeconduct.zoom) (1.5 by default), and hides the rest of the page; unset, the page is shown as it is.

Tips

At the console, in an interactive session, once a host has been found to need a window or the browser for the transfer, a tip says how to skip the finding out next time: headed = TRUE, or a browser_session() for several files or reads from the host. A tip is shown once per host and R session, and never when quiet is TRUE. Calls made from another package get no tips, since that package's user cannot act on them; nothing needs to be set for that. The tips are messages with the class safeconduct_hint as well, should anyone want to muffle them.

Examples

if (FALSE) { # \dontrun{
url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/",
              "geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip")
fetch(url, "lrnf000r21a_e.zip")
} # }