Skip to contents

Read a resource into memory, through a browser if the server insists

Source: R/fetch.R
read_page.Rd

The counterpart of fetch() for a page or a feed rather than a file: the response is returned instead of written to disk, in the shape of curl::curl_fetch_memory()'s. The same path is taken: curl where a plain request is served, curl carrying the browser's clearance where the server accepts that, and the browser itself where it does not, in which case the resource is read by the page the browser holds on the host.

Usage

read_page(
  url,
  session = NULL,
  quiet = FALSE,
  timeout = 120,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

session

A browser session from browser_session() on the resource's host. When given, the resource is read in that browser without any probing.

quiet

Suppress progress messages.

timeout

Seconds a read in the browser may take.

headed

How to run the browser, when one is needed: "auto", the default, tries headless Chrome first and opens a window only when the headless run does not get through, so that a window is shown only when someone may need to click; TRUE opens a window at once and FALSE never does. Without a display "auto" is headless. A host may run a second check at the file's URL, after the one at its root; headless Chrome has passed such a check with nothing clicked, but only after about two minutes, where a window shows a box that gets it through as soon as it is clicked. "auto" waits those minutes and opens a window if they pass without; TRUE skips the wait.

useragent

The user-agent string the browser announces. Headless Chrome's own calls it HeadlessChrome, and some hosts refuse it on that word alone. TRUE, the default, drops the word, so that the string says the Chrome version and platform it is and nothing it is not; NULL leaves Chrome's own string; a string is used as given. Set before the browser's first request, and what clearance() then reports.

Value

A list: url (the final URL), status_code, type (the content-type, lower case), headers (a named list) and content (raw). An HTTP error status is returned, not raised; content is then the error page. Errors are classed as fetch()'s are. Parse it with page_text(), page_json() or page_html().

Details

Without a session, a browser is opened and closed per call. For a feed that is read every few minutes, open a browser_session() once and pass it: the resource is then read in that browser, and if the server asks the browser for a check again meanwhile, the check is passed and the read repeated.

Examples

if (FALSE) { # \dontrun{
r <- read_page("https://results.vancouver.ca/data/arearesults.json")
page_json(r)
} # }