Pull latest data from external scraper repos and report coverage

Fast-forward git pulls each of the five external scraper repos that MOSAIC depends on for raw data, then prints a per-repo summary of the latest commit and (for surveillance repos) the date range of the data file the process_*() functions read. Designed to be called at the top of model/LAUNCH.R so a refresh run starts from a known-current snapshot.

Usage

refresh_data_repos(root = NULL, repos = NULL, stale_days = 14L, verbose = TRUE)

Arguments

root: Path to the MOSAIC parent directory (containing all sibling repos). Defaults to get_paths()$ROOT, i.e. whatever set_root_directory() was last called with.
repos: Optional character vector restricting the refresh to a subset of repos. Valid names: "ees-cholera-mapping", "jhu_cholera_data", "ai-cholera-data-mining", "enso-data", "open-meteo-pipeline". Default refreshes all five.
stale_days: Threshold (days since last commit) for the STALE flag in the printed summary. Default 14.
verbose: If TRUE (default), print a formatted summary to the console.

Value

Invisibly, a list of per-repo results with fields repo, description, ok (pull succeeded), last_commit_date, last_commit_msg, days_stale, coverage (named list of coverage stats for surveillance repos; NULL otherwise), and pull_output.

Details

Uses git pull --ff-only so the call is non-destructive: a divergent local branch will fail loudly rather than auto-merge. Uncommitted local changes will also block the pull and surface in pull_output.

For ees-cholera-mapping the function additionally peeks at the WHO AWD weekly file process_WHO_weekly_data() reads and reports record count, date range (week-start Mondays derived from WHO's year/week) and country count.

JHU is a static archive. process_JHU_weekly_data() does NOT read the jhu_cholera_data repo: it reads the frozen OSF archive MOSAIC-data/raw/JHU/osfstorage-archive/Public_surveillance_dataset.rds. Pulling the repo therefore never changes MOSAIC's JHU input; the summary reports the archive's year range and file date, labelled as static. ENSO and open-meteo coverage stats are not extracted because their data is spread across many per-country / per-source files; commit date is the practical freshness signal.

Examples

if (FALSE) { # \dontrun{
set_root_directory("~/MOSAIC")
refresh_data_repos()

# Only the surveillance scrapers
refresh_data_repos(repos = c("ees-cholera-mapping", "jhu_cholera_data"))
} # }