Fetches url to destfile. A plain request is made first, and if the
file comes no browser is involved. If the server answers with a browser
check (is_challenge()) or a block page (is_block()), a Chrome with a
throwaway profile is opened on the host, any check runs, and the file is
downloaded: through curl carrying the browser's clearance where the server
accepts that, and in the browser itself where it does not. A clearance
curl can use is kept for the rest of the R session, so later files from
the host need no browser.
Usage
fetch(
url,
destfile,
quiet = FALSE,
timeout = 3600,
session = NULL,
headed = "auto",
useragent = TRUE
)Arguments
- url
The file's URL.
- destfile
Where to write it. A failed transfer leaves nothing there.
- quiet
Suppress progress messages.
- timeout
Seconds a transfer in the browser may take.
- session
A
browser_session(). The file is then downloaded in that browser without probing, with the session'sheadedanduseragent, and the browser stays open; one that has gone away is reopened first.- headed
How to run the browser when one is needed.
"auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on;TRUEopens a window at once;FALSEnever does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked:"auto"waits those minutes before opening a window,TRUEskips the wait.- useragent
The browser's user-agent string.
TRUE, the default, drops the wordHeadlessfrom Chrome's own string, which some hosts refuse on that word alone, and changes nothing else;NULLleaves Chrome's own; a string is used as given.
Value
destfile, invisibly. Errors are classed under
safeconduct_error: safeconduct_unavailable when no browser can be
opened here, safeconduct_not_passed when the check was not passed in
time or the server blocks the browser too, with a reason field, and
safeconduct_download_error when the transfer failed.
Details
Nothing is simulated. A check that shows a "Verify you are human" box
waits in a window, for up to two minutes, for a person to click it. The
window is brought to the front when a check shows in it, and wants to be
left there until it is through: a check run with the window behind
something else is reissued on every click. A
file downloaded in the browser lands under tempdir() and is moved to
destfile when complete.
Requirements
Chrome or Chromium, found by chromote::find_chrome(); the
CHROMOTE_CHROME environment variable points at one somewhere unusual,
and browser_available() says whether one can be opened here.
options(safeconduct.browser = FALSE) never opens a browser; the window's
size and the other options are described in safeconduct-package.
Tips
At the console, once a host has been found to need a window or the browser
for the transfer, a tip says how to skip the finding out next time:
headed = TRUE, or a browser_session(). Tips come once per host and R
session, not when quiet, and not for calls made from another package.
They are messages of class safeconduct_hint.
Examples
if (FALSE) { # \dontrun{
url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/",
"geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip")
fetch(url, "lrnf000r21a_e.zip")
} # }
