The mathematics of model calibration
This section develops the principles of model calibration from a statistical perspective. Core topics covered include problem formulation, goodness-of-fit (likelihood) functions, and a comparison of optimization and posterior sampling approaches.
Model formalism and problem formulation
Disease models can be viewed as complex functions that map input parameters, \(\theta\), to state trajectories, \(X\), and observations \(y\). While some models are deterministic, meaning they produce the same state trajectory and output for a given input every time, others are stochastic. Agent-based models, implemented using frameworks like Starsim, EMOD, and LASER, are fundamentally stochastic by design; each set of input parameters, \(\theta\), can yield many different state trajectories and outputs, yet the outputs are reproducible given the seed of the random number generator.
The parameters, \(\theta\), can include biological parameters (e.g., transmission rates, incubation periods), behavioral parameters (e.g., contact rates, intervention adherence), intervention parameters (e.g., vaccination coverage, treatment efficacy), and other factors that influence disease dynamics. Initial conditions could also be included in the parameters.
The state trajectory, \(X\), represents the evolution of the disease over time. For a susceptible-infected-recovered (SIR) model, for example, the state trajectory would include the number of susceptible, infected, and recovered individuals at each time step. For an agent-based model, the state \(X\) is very high dimensional as it includes every detail about every individual in the simulation.
Many models of health and disease assume Markov dynamics: the state at time \(t+\Delta t\) depends only on the state at time \(t\) and the parameters \(\theta\). The state trajectory is a sequence over discrete time steps, \(X=(X_0, X_{\Delta t}, \ldots, X_T)\), where \(X_t\) is the state at time \(t\) and \(T\) is the final time. Formally, we write the state transition as:
The state \(X\) is latent, meaning it cannot be directly observed.
The observations, \(Y\), are random variables linked to the state by a measurement (observation) model and represent quantities we can measure in practice. Examples include daily or cumulative case counts, hospitalizations, deaths, seroprevalence, or other epidemiological metrics. Observation models should account for reporting rates, delays, and other biases. We write:
As part of model calibration, we will be comparing model-based realizations \(\tilde y_t^{(r)}\) of \(Y_t\) to real-world observations, denoted \(y_t^{\text{obs}}\).
Finally, note that for deterministic models, the conditional probability distributions described in this section collapse to point masses.
Notation
| Symbol | Description | Type |
|---|---|---|
| \(\theta\) | Model input parameters | Fixed |
| \(X_t\) | Model state at time \(t\) | Random variable |
| \(\tilde{x}_t^{(r)}\) | Model state at time \(t\) for realization \(r\) | Realization from simulator |
| \(Y_t\) | Model output (observation) at time \(t\) | Random variable |
| \(\tilde{y}_t^{(r)}\) | Simulated data at time \(t\) for realization \(r\) | Realization from simulator |
| \(y_t^{\text{obs}}\) | Measured data at time \(t\) | Real-world observation |
At times, we replace the single time index \(t\) with a range. For example, \(\tilde x_{0:T}^{(r)}\) denotes a simulated state trajectory for the \(r^{th}\) realization from time \(0\) to \(T\) (similarly \(\tilde y_{0:T}^{(r)}\) and \(y_{0:T}^{\text{obs}}\)). Other times, we drop the time index entirely to refer to the whole trajectory, e.g., \(\tilde{x}^{(r)}\).
When discussing multiple realizations, we use \(r=1,\ldots,N\) to index the different runs of the simulator. For example, \(\tilde{y}_t^{(1)}, \ldots, \tilde{y}_t^{(N)}\) are \(N\) independent realizations of the output at time \(t\). We often drop the superscript when referring to a single realization, e.g., \(\tilde{y}_t\).
Model fit for purpose
Models come in all shapes and sizes. Simple models can be built from scratch in a few lines of code, while complex agent-based models are best implemented using specialized frameworks like Starsim, EMOD, or LASER. Before starting a modeling analysis, ask yourself the following questions in relation to the research or policy question at hand:
- What scientific features are necessary to get the right answer?
- What type of model (statistical, compartmental, agent-based) is best suited to the question?
- What unknowns could affect the answer?
- How precise does the answer need to be?
In addition to building/adapting a model for each research question, we must also consider the epidemiological context in which the model will be applied. A model that works well for one setting may not be appropriate for another due to differences in demographics and mobility patterns, health infrastructure and delivery channels, social behavior, and other factors.
An idealized research flow is:
- Define the modeling question, decisions, success metrics, and validation plan. Think about what "good enough" means in terms of fidelity and uncertainty quantification.
- Gather & preprocess data; create splits (temporal/spatial holdout) before peeking. Keep an untouched external/test set (or identify an external dataset) for the final check.
- Select a modeling framework that is fit for purpose.
- Implement candidate model(s) including a simple baseline model.
- Calibrate (fit) on the training data using priors as needed.
- Perform internal validation & report diagnostics; iterate. Use residual analysis and posterior predictive checks to diagnose in-sample misfit and guide improvements. Use time-respecting cross-validation (e.g., rolling-origin for time series) for out-of-sample evaluation.
- Perform external validation on independent data (single shot). Do not use this data for tuning. If it fails here, go back to steps 3–6.
- Use the calibrated model to address the original modeling question, report findings with uncertainty, compute information value if needed.
Often we do not know a priori which dynamics and factors are most important to include in a contextualized model. Therefore, we may need to consider multiple candidate models and use data to help select between them. Each candidate structure may have many unknown parameters that must be considered for sound inference. These considerations lead us to the topic of model calibration, step 5 above.
Model calibration
Model calibration is the process of fully contextualizing an analysis to a region and/or population of interest as part of addressing a modeling question. Calibration should not be seen as a one-time "push button" activity, but rather as an iterative process of model refinement and learning.
The joy of calibration is that we get to confront our models with real-world data. Conditioning on data allows us to learn. We can use data to select between candidate model structures, estimate model parameters, uncover the latent model state, determine if more information is needed, and ultimately provide robust support to decision-makers.
Therefore, a key aspect of model calibration is the data, here denoted \(y_{0:T}^{\text{obs}}\). The data can come from a variety of sources, including surveillance systems, clinical studies, serological surveys, and more. The data may be noisy, incomplete, or biased; issues that must be considered when designing the model and calibration approach. The model should include an observation model intended to capture the data generation process, but residual imperfections in modeled dynamics or observation processes can be accounted for in calibration discrepancy terms.
Note
The goal of calibration is to make outputs from the model(s) look like data from the real world to support decision making. For deterministic models, calibration entails finding one or more sets of model input parameters \(\theta^{(m)}\) for \(m=1,\ldots,M\) such that simulation output \(y(\theta^{(m)})\) is close to the observed data \(y^{\text{obs}}\). For these models, the latent state trajectory \(X\) is fully determined by the input parameters, so calibration focuses exclusively on parameter estimation.
For stochastic models, calibration requires jointly identifying pairs of input parameters and latent state trajectories. The result of calibration is a pair or set of pairs of the form \((\theta_m, \tilde x_{0:T}^{(m)})\) for \(m=1,\ldots,M\) that yield model output realizations \(\tilde y^{(m)}\) that are close to the observed data \(y^{\text{obs}}\). The latent state trajectory is important because it reflects the current state of the world, which is often not directly measured.