cansim 0.4.5
Major changes
StatCan being unavailable no longer aborts with an error. Timeouts, connection failures and error responses are now reported with a loud warning and the function returns
NULL, so that a script or a document can decide for itself what to do when the servers are down. This applies to every function that talks to StatCan, and it also covers the two calls that previously bypassed the retry helper,get_cansim_table_last_release_date()andget_cansim_series_info_cube_coord(). Setoptions(cansim.error_on_unavailable=TRUE)to get the previous behaviour of raising an errorexamples that make a single lightweight API call are now
\donttest{}rather than\dontrun{}, so they are checked rather than merely displayed. Examples that download a full table or the cube list stay\dontrun{}because of their run time, andcansim_old_to_new()needs no network at all so its example now always runsdata retrieved by vector or by table/coordinate now carries
UOMandUOM_IDcolumns, taken from the cube metadata. StatCan flags a single dimension of each cube as carrying the unit of measure and the unit varies by member of that dimension, so the unit is resolved per coordinate. Tables that have no unit of measure, for example census tables, get no unit columns, matching the full table download (#170). With the unit known, percentage values retrieved by vector or by coordinate are now normalized the same way as full table downloads:val_normcarries the value divided by 100 and the unit of measure is relabelled toRate(Tauxin French). Previously the same series normalized differently depending on whether it was retrieved as a full table or by vector, so scripts that fetch percentage vectors will seeval_normchange by a factor of 100non-breaking spaces and control characters in names returned by StatCan are now replaced with regular spaces. These characters render as an ordinary space or as nothing at all, so a column whose name contained one could not be reached by typing or copy-pasting what the console displayed. The repair covers table downloads, vector and coordinate calls, cube metadata, table templates and the cube list, and emits a warning that shows the offending characters by code point, for example
Performance<U+00A0> strategy, together with a count of how many names were repaired. Setoptions(cansim.suppress_repair_warnings=TRUE)to silence the warning. The warning also says that the characters are in the data StatCan publishes rather than anything the user did, and that it will disappear on its own once StatCan stops sending them, pointing at the issue where that is tracked. Column names of tables cached before this release keep the original characters until the table is downloaded again,get_cansim_connection()warns when it finds such a cache (#169)the same repair now also covers the member labels in the data itself, not just the names of the columns holding them. These characters turn out to be more common in member labels than in dimension names, 53 of 500 sampled tables carry at least one. Repairing only the metadata side would have left the labels in the data unable to match their own factor levels, so every row carrying an affected label would have become
NA. Labels are now also identical whichever way the data is retrieved, so a table can be joined to template, vector or coordinate data on its dimension columns (#169)an internal
scan_statcan_character_problems()reads the cube metadata straight off the API, without the repair applied, and reports every title, dimension name and member name StatCan publishes with a non-breaking space or a control character in it, by table, level and language.summarize_statcan_character_problems()aggregates that by survey. Neither is exported, they exist to track whether the upstream problem is shrinking, and both go away along with the repair once it has (#169)cached tables now record the package version they were parsed under alongside the download timestamp, in a single
.Rda_infofile that replaces the.Rda_timefile the timestamp used to have to itself. The timestamp says whether StatCan has newer data, the version says whether this release still reads those files the same way.list_cansim_cached_tables()reports it in a newcansimVersioncolumn, empty for anything cached before this release. The old timestamp file is still read, so existing caches keep their download date, and is replaced when a table is refreshed.get_cansim_connection()uses the version to check whether a cache predates the repair of non-breaking spaces and control characters, and if so reads the metadata cached alongside the table to see whether its dimension names or member labels actually carry any. Only then does it warn, naming the offending label and pointing atrefresh=TRUE(#169)every call that sends StatCan a list of vectors, coordinates or tables is now split into batches of at most 300 items. StatCan refuses a longer list outright with an HTTP 416, which had gone unnoticed because most of these calls already batched.
get_cansim_vector_info()and the cube metadata download did not, so asking either for more than 300 items at a time failed rather than returning dataa vector or coordinate StatCan cannot answer for is no longer passed off as data. StatCan signals a bad item two different ways depending on the method, either marking the record
FAILEDor answeringSUCCESSand putting the reason inresponseStatusCode, and the package only checked the first. That let an invalid vector throughget_cansim_vector_info()as a row ofNAs indistinguishable from real metadata. Both are now checked everywhere, and the items that carry no data are dropped and reported by reason, naming the vectors or coordinates concernedvector calls that come back with nothing now warn and return an empty table. Previously the empty answer travelled on to the metadata join and surfaced there as
Column 'cansimTableNumber' doesn't exist, which said nothing about what had happened. The warning names all three things that produce it: vectors that do not exist, vectors with no data in the requested time frame, and the daily window from midnight to 8:30am Eastern in which StatCan does not serve vector datawhen StatCan refuses a request it explains why in the response body, and that explanation is now shown alongside the status code instead of being discarded. An HTTP 409 says whether the product is simply not released yet, and an HTTP 416 names the limit the request went past. Those two status codes also got the plain-language translation the other codes already had. The HTTP 504 message now says that StatCan builds a whole response before sending any of it, so the way past a gateway timeout is to ask for less at once rather than to retry the same request
three new functions expose StatCan’s changed series methods, which report what changed at a finer grain than
get_cansim_changed_tables()does.get_cansim_changed_series_list()lists the series StatCan changed today as vectors, with the table and coordinate each belongs to, andget_cansim_changed_series_data_for_vectors()andget_cansim_changed_series_data_for_coordinates()retrieve the changed data points themselves, in the same shape and with the same metadata as the correspondingget_cansim_vector()and coordinate calls. Series that did not change simply contribute no rows, and if none of the ones asked about changed the answer is an empty table rather than an error. Like the other vector methods the two data ones batch requests of more than 300 items. How long the list method takes is entirely a matter of how much StatCan released that morning. It takes no parameters, so a busy day cannot be asked about in smaller pieces, and on one heavy enough the request has been seen to outlive StatCan’s own gateway and come back as an HTTP 504 after some nine minutes of silence. Itstimeoutdefaults high to let that answer arrive as StatCan’s own rather than as a vaguer local abort, but the limit is at StatCan’s end and raising it further will not help;get_cansim_changed_tables()is the question to ask on such a daythe package now talks to StatCan through
httr2rather thanhttr. Requests that fail on a status StatCan recovers from within seconds, an HTTP 429, 500, 502 or 504, are now retried with exponential backoff and jitter and honour aRetry-Afterheader, where the previous retries went out back to back and stood a good chance of arriving while the server was still busy. Requests are also throttled to the 25 per second StatCan documents as its per-IP limit, so a script asking for many tables or vectors no longer risks being turned away for asking too quickly, and they now identify themselves with acansim/<version>user agent. Statuses that will not improve on a retry are deliberately not retried: an HTTP 416 carries more items than StatCan accepts however often it is sent, an HTTP 409 is the nightly update window, and an HTTP 503 is StatCan being down for maintenance or an outage, which lasts far longer than any retry budget worth spending. The 503 case says so, and says to try again later, rather than appearing to hang while retries run down. Users behind a proxy should note thathttr2reads the standardhttp_proxyandhttps_proxyenvironment variables instead of taking anhttr::set_config()callthe
timeoutargument now bounds how long StatCan may go without sending anything, rather than how long the whole transfer may take. As a cap on the total it could not tell a connection StatCan had stopped answering on from a large table that was simply taking a while, and cut both off alike, so a slow download could fail after two hundred seconds with most of the data already in hand. A transfer that keeps delivering is now left alone however long it runs, and one that goes quiet fortimeoutseconds is dropped, which is the distinction the argument was always described as making. Note that StatCan works out a whole response before sending any of it, taking roughly a tenth of a second per vector, so a request for a full batch of 300 is silent for something like thirty five seconds before the first byte arrives; the default of two hundred seconds leaves ample room for that, but a much smaller value passed by hand will cut off large requests. Establishing the connection is bounded separately and briefly, so an unreachable host now fails in ten seconds instead of waiting out the full timeoutget_cansim()andget_cansim_connection()now ask StatCan where a table lives instead of assembling the download address from the table number. The address the package built was a guess at a layout StatCan is free to change, and the extra call that replaces it is small next to the table download it precedes
Deprecations
-
get_cansim_sqlite(),list_cansim_sqlite_cached_tables()andremove_cansim_sqlite_cached_table()are now also documented as deprecated, matching the deprecation warnings they already emit. Useget_cansim_connection(..., format="sqlite"),list_cansim_cached_tables()andremove_cansim_cached_tables(..., format="sqlite")instead -
disconnect_cansim_sqlite()is deprecated in favour of the newdisconnect_cansim_connection(), which does the same thing under a name that does not claim a format. It closes a sqlite connection and leaves parquet and feather connections alone, so a connection can be closed without knowing which format it came from. It was the last function still named for sqlite that was not itself deprecated, and its own example demonstrated the deprecatedget_cansim_sqlite() - the deprecated
get_cansim_sqlite(),list_cansim_sqlite_cached_tables(),remove_cansim_sqlite_cached_table(),disconnect_cansim_sqlite(),list_cansim_tables()andsearch_cansim_tables()are scheduled for removal in a future release
Performance
- vector queries now collect their 300-vector API batches in a list and combine them once, instead of repeatedly copying all previously collected rows into every new batch. In an offline warm-cache benchmark, combining 100 batches and 150,000 rows improved from 0.424s to 0.227s
- hierarchy building in metadata parsing no longer re-parses the growing hierarchy paths, hierarchies are built one ancestor level at a time across all members at once
- coordinates are split once into a character matrix when folding in metadata and converting to factors
- factor conversion of dimensions with duplicate member names splits only the unique coordinates instead of every row, a table repeats each coordinate once per reference period. On 36-10-0580 this is 6,882 unique coordinates against 996,978 rows, cutting a cached read from 4.5s to 4.0s
- table templates are built with a single cartesian product instead of joining one dimension at a time
- metadata for data retrieved by vector or by table/coordinate is now resolved for all coordinates at once. The member table of each dimension used to be rebuilt for every single coordinate, which made this step grow linearly at about 24ms per coordinate. Resolving 200 coordinates of 36-10-0580 went from 5.0s to 0.02s, and all 10,164 coordinates of that table now take 0.03s. Warnings about members missing from the cube metadata are reported once per member rather than once per coordinate that uses it
Minor changes
- asking
get_cansim_vector_for_latest_periods()orget_cansim_data_for_table_coord_periods()for all periods no longer sends an arbitrary round number as the period count. StatCan takeslatestNas a signed 32-bit integer, rejecting zero or less and anything past 2147483647, and quietly clamps a count longer than the series to the whole series, so the default is now that bound as the API itself enforces it, rather than a guess that happened to exceed the longest series. A period count larger than the bound, or an infinite one, is capped instead of being silently coerced toNAand sent to StatCan as"latestN":NA, and a count below one now fails immediately with a message instead of earning an HTTP 406. This also applies to the per-coordinateperiodscolumn of a table template - an unrecognized
languageargument is now an error naming what was passed, instead of anNAthat travelled on into a cache directory name or the tail of a StatCan URL and surfaced later as a download failure or a missing column. Either language can be named in either language, so"english","en","eng"and"anglais"all select English and"french","fr","fra"and"français"all select French, along with their longer and shorter forms; case, surrounding whitespace and accents are ignored.get_cansim_table_url()andget_cansim_table_notes()now default to"english"like every other function that takes a language, which selects the same language their previous"en"default did (#152) - drop the unreachable
if (TRUE) ... else ...in metadata parsing. The else branch held thereadr::read_delim()implementation thatutils::read.delim()replaced in February 2025 and had since fallen behind the live branch, so it was no longer a working fallback (#151) - fix a
case_when()deprecation warning emitted by dplyr 1.2.0 on every table read - fix
get_cansim_changed_tables()passing “days” todifftime()as a time zone instead of a unit -
get_cansim_changed_tables()now takes both the current date and the cutoff after which the day’s changes are available in Eastern time. It used to compare against 9am, half an hour after StatCan actually closes its nightly update window, and to take “today” from the local clock, so a machine set west of Eastern could ask StatCan about a day that had not started there yet -
get_cansim_connection()no longer fails when the release date of a table cannot be determined, the staleness check is skipped with a message instead - the unit of measure columns of French language tables are now ordered with the other value columns, as they already were in English language tables
- better connection error handling
- fix
get_cansim_cube_metadata()andget_cansim_table_template()for vectors of table numbers, metadata for all tables is still retrieved in a single API call and cached per table -
get_cansim_cube_metadata()adds acansimTableNumbercolumn for the “members”, “notes” and “corrections” types - functions that only operate on a single table now fail with an informative message when given several table numbers
- normalizing percentages now relabels the unit of measure to
Rate, orTauxin French tables, as the documentation always described; the comparison doing the relabelling could never match before, so the unit columns used to keep their originalPercent...labels after the values had been divided by 100 -
add_cansim_vectors_to_template()now finds vectors for coordinates whose member ids end in a zero, trimming of trailing.0positions used to eat into member ids like10and the affected rows came back with anNAvector - French connections no longer emit a spurious “Unknown table type” warning on
collect_and_normalize(), an internal language comparison never matched the French setting - when refreshing a cached table fails because StatCan is unavailable,
get_cansim_connection()now falls back to the previously cached version with a warning instead of returningNULL. Refreshing cube metadata degrades the same way, so the notes, column, overview and template functions keep working from previously seen metadata when the servers are down - the duplicated-column error when caching a table now actually names the offending columns and no longer blames SQLite for parquet and feather connections
cansim 0.4.2
CRAN release: 2025-05-12
Minor changes
- ensure proper ordering of levels even if StatCan metadata is not ordered
- better error messages and information on how to disable peer checking when StatCan SSL certificates have problems
- automatically batch vector or coordinate data retrieval in case users request more than 300 series at a time ## Major changes
- enable series information by table and coordinate
- generate table template and facilitate adding vector info to aid pinpointed data download
- enable downloading of data by vector and multiple coordinates in get_cansim_data_for_table_coord_periods (breaking changes with change to parameter)
cansim 0.4
CRAN release: 2025-02-24
Major changes
- add support for local caching in parquet and feather formats
- uniform interface for sqlite, parquet, and feather caching
- principled approach to column order ## Minor changes
- fix problem with inconsistent type parsing of notes
- better support for french language when accessing data by vector or coordinate
- tests
cansim 0.3.17
CRAN release: 2024-11-06
Minor changes
- fix problem with reading French tables released by the census division
- restore original column order after converting to factors
- convert geography column to factor if available
- fix problem with
add_provincial_abbreviationsthat could lead to mislabelling of provinces in some cases - improve handling of metadata, enable downloading only metadata instead of only via full table download
- fold metadata into data when accessing via vector or coordinates
- allow for cansim vectors in
view_cansim_webpageto view vector information from statcan in the browser
cansim 0.3.13
CRAN release: 2022-11-07
Minor changes
- Speed up access to cached sqlite tables
- Fix problem with
get_cansim_vector_info()
cansim 0.3.9
CRAN release: 2021-07-29
Major changes
- deprecate
list_cansim_tablesandserach_cansim_tablesand fallback to corresponding “_cube” methods as Open Data Canada API has changed and similar functionality is available through the “_cube” methods that tie directly into StatCan APIS ## Minor changes - Fix issues with top level duplicate categories
- Check for expired tables in
list_cansim_sqlite_cached_tables - New auto-update feature for sqlite tables
cansim 0.3.8
CRAN release: 2021-05-27
Minor changes
- Exclude vignette from automatic CRAN checks to fix problem of CRAN checks failing when StatCan servers are down and lead to the package being removed from CRAN (checks are still active in local environment and when using GitHub action checks)
- add release date info to cube metadata and cube list calls
- add auto-refresh option for sqlite tables
- remove deprecated
adjust_cansim_values_by_variablefunction
cansim 0.3.6
CRAN release: 2021-05-08
Major changes
- Fold part of
normalize_cansim_valuesinto the default table and vector output, in particular always add a scaled variable column calledval_normand an imputedDatecolumn and covert categories to factors by default. - New
get_cansim_sqlitefunction that stores tables in an SQLite database and facilitates access and management of data.
cansim 0.3.4
CRAN release: 2020-03-05
Minor changes
- Expand
get_cansim_table_notes()functionality - Add functionality to access the new cube list API
cansim 0.3.0
CRAN release: 2019-07-18
Minor changes
- Fixes issues arising from StatCan changing their API
- Member Names come concatenated with Classification Code by default, this could break existing code.
- Adds option to change fields to factors
- Adds option to strip Classification Codes from fields
- Exposes timeout limit to deal with slow connections and large tables