Skip to contents

Fetches url to destfile. A plain request is made first, and if the file comes no browser is involved. If the server answers with a browser check (is_challenge()) or a block page (is_block()), a Chrome with a throwaway profile is opened on the host, any check runs, and the file is downloaded: through curl carrying the browser's clearance where the server accepts that, and in the browser itself where it does not. A clearance curl can use is kept for the rest of the R session, so later files from the host need no browser.

Usage

fetch(
  url,
  destfile,
  quiet = FALSE,
  timeout = 3600,
  session = NULL,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

destfile

Where to write it. A failed transfer leaves nothing there.

quiet

Suppress progress messages.

timeout

Seconds a transfer in the browser may take.

session

A browser_session(). The file is then downloaded in that browser without probing, with the session's headed and useragent, and the browser stays open; one that has gone away is reopened first.

headed

How to run the browser when one is needed. "auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on; TRUE opens a window at once; FALSE never does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked: "auto" waits those minutes before opening a window, TRUE skips the wait.

useragent

The browser's user-agent string. TRUE, the default, drops the word Headless from Chrome's own string, which some hosts refuse on that word alone, and changes nothing else; NULL leaves Chrome's own; a string is used as given.

Value

destfile, invisibly. Errors are classed under safeconduct_error: safeconduct_unavailable when no browser can be opened here, safeconduct_not_passed when the check was not passed in time or the server blocks the browser too, with a reason field, and safeconduct_download_error when the transfer failed.

Details

Nothing is simulated. A check that shows a "Verify you are human" box waits in a window, for up to two minutes, for a person to click it. The window is brought to the front when a check shows in it, and wants to be left there until it is through: a check run with the window behind something else is reissued on every click. A file downloaded in the browser lands under tempdir() and is moved to destfile when complete.

Requirements

Chrome or Chromium, found by chromote::find_chrome(); the CHROMOTE_CHROME environment variable points at one somewhere unusual, and browser_available() says whether one can be opened here. options(safeconduct.browser = FALSE) never opens a browser; the window's size and the other options are described in safeconduct-package.

Tips

At the console, once a host has been found to need a window or the browser for the transfer, a tip says how to skip the finding out next time: headed = TRUE, or a browser_session(). Tips come once per host and R session, not when quiet, and not for calls made from another package. They are messages of class safeconduct_hint.

Examples

if (FALSE) { # \dontrun{
url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/",
              "geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip")
fetch(url, "lrnf000r21a_e.zip")
} # }