Post-hoc evaluation of a rolling-window forecast-validation artifact

Reads a run_rolling_cv predictions artifact and computes out-of-sample forecast skill: per-cell metrics (R2 correlation & SSE, bias ratio, 50/95% PI coverage, WIS) for the in-sample fit and for cumulative OOS horizons, plus MAE-skill and WIS-skill versus seasonal-climatology and persistence baselines, aggregated across cutoffs and countries with bootstrap CIs (suppressed at small cell counts).

Usage

evaluate_rolling_cv(
  predictions,
  horizons_months = c(1, 3, 5),
  baselines = c("seasonal", "persistence", "persistence_last"),
  metrics = c("cases", "deaths"),
  trusted_only = TRUE,
  embargo_weeks = 0,
  ess_min = 0,
  min_cells_ci = 5L,
  n_boot = 1000L,
  seed = 1L,
  observed_tiers = c(1, 2)
)

Arguments

predictions

A run_rolling_cv output directory, a path to a predictions.parquet, or the predictions data frame itself.

horizons_months

Cumulative OOS horizons in months (default c(1,3,5)).

baselines

Baselines for skill scores (default c("seasonal","persistence","persistence_last")).

metrics

Metrics to evaluate (default c("cases","deaths")).

trusted_only

Drop rows with observed_source == "AI" (default TRUE).

embargo_weeks

Channel-specific OOS embargo, in weeks, after the cutoff before scoring begins, applied on top of the harness embargo (see Details). A scalar applies to all metrics; a named vector (e.g. c(cases = 4, deaths = 6)) applies per metric, and omitted metrics get 0. Default 0.

ess_min

Turns the convergence gate on when > 0: cells whose cutoff run is red (or, before MOSAIC 1.4.0, whose ESS is below ess_min) are left out of the summary. Default 0 (no gate). See Details.

min_cells_ci

Minimum number of finite cells in a (model, metric, window) group required to emit a bootstrap CI; below this the CI is suppressed (lo = hi = NA) and ci_suppressed = TRUE. Default 5.

n_boot

Bootstrap resamples over cells for the skill CI (default 1000).

seed

RNG seed for the bootstrap (default 1).

observed_tiers

Surveillance tiers scored out of sample when the predictions carry observed_tier (default c(1, 2): observed and reconstructed weeks); NULL scores every tier. See Details.

When the predictions carry a model column (ensemble, ensemble_opt, best, medoid), every metric is computed and reported per model; absent that column all rows are treated as a single "ensemble" model (back-compatible).

Value

A list with:

cells

Per (cutoff x model x iso x metric x window) metrics + per-baseline mae_skill and wis_skill, the cell ess, convergence_status and ess_ok flag (plus anchor_date when the predictions carry it).

summary

Aggregated across ess_ok cells per (model x metric x window): n_cells_used / n_cells_total, mean/median of each metric, and mean baseline skills with bootstrap CI (suppressed when fewer than min_cells_ci finite cells), plus ess_gated and ci_suppressed flags.

Details

This is the post-hoc companion to run_rolling_cv() — the harness emits predictions only; all judgement (metrics, baselines, skill) lives here.

Horizons are cumulative (\(\le h\) months): the \(\le h\) window is every scored OOS date \(d\) with origin < d <= origin + ceiling(h * 30.4375) days, where origin is the per-metric OOS-scoring origin below. The origin is fixed by the cutoff and embargo, never by where observations happen to exist, so a reporting gap just after the cutoff shrinks a window's n instead of shifting its lead times (windows with fewer than 3 scorable dates are dropped). The \(\le\)max-horizon window is the largest scored set; predictions further out are not scored in any window. An IS window (training fit, \(\le\) cutoff) is always reported.

Scoring uses trusted observed only: rows whose observed_source is missing or not "AI" (when trusted_only = TRUE).

Surveillance tier. When the predictions carry observed_tier (the config's reported_tier: 1 observed, 2 reconstructed, 3 imputed), out-of-sample days whose tier is not in observed_tiers are not scored. The in-sample rows, which the baselines are fitted on, keep every tier, as the calibration did. Predictions without the column are scored unchanged.

Embargo and scoring origin. The harness labels the dates in (cutoff, cutoff + embargo] as segment == "embargo"; those are never scored. For each metric the scoring origin is max(last embargo-labelled date (or the cutoff), cutoff_date + embargo_weeks[metric] * 7) and the scored dates are those strictly after it. embargo_weeks can therefore only lengthen the harness embargo, per channel: it may be a scalar (all metrics) or a named vector such as c(cases = 4, deaths = 6); metrics it omits get 0. With the default (0) exactly the segment == "OOS" block is scored, from the harness's own OOS origin.

Baselines, fit per (cutoff, country, metric) on the IS observed history. "seasonal" (week-of-year climatology) is the primary baseline and requires \(\ge 2\) years of in-sample history; with less, the seasonal baseline (and its skill) is NA (no grand-mean fallback). "persistence" (trailing 4-point mean carried forward) and "persistence_last" (the single last observed value) are secondary nulls. Each baseline is also given 50/95% predictive intervals from empirical quantiles of its in-sample residuals, so a genuine WIS-skill exists: \(1 - WIS_{model}/WIS_{baseline}\). mae_skill is the analogous MAE ratio. Positive = model beats the baseline; NA when the baseline (or its interval) cannot be formed.

Convergence gate (opt-in). With ess_min = 0 (the default) there is no gate: every cell has ess_ok = TRUE and enters $summary. With ess_min > 0 the gate is on: a cell whose predictions carry a convergence_status (MOSAIC 1.4.0: the cutoff calibration's simulation-convergence status, constant per cutoff) passes when the status is not "red"; a cell without one (a cutoff run from before 1.4.0) falls back to the weight-ESS rule ess_ok = (!is.na(ess) & ess >= ess_min); a cell with neither passes. Cells failing the gate remain in $cells (flagged) but are excluded from the $summary aggregation. $cells$convergence_status is carried as information whatever ess_min is. A warning is issued when more than half of the cells that carry a status are red (cheap or short cutoff calibrations), and when the gate removes every cell.

See also

run_rolling_cv, calc_model_R2, calc_bias_ratio