Process best practices
This topic describes best practices for the calibration techniques themselves. Calibrating a disease model can be frustrating. See the mathematics of model calibration for the statistical principles around calibration. As you approach the process, consider the following suggestions:
Know your data
Gather, clean, and fully understand the calibration data. Which model outputs will align with the data? (Derive new model outputs, if needed.) How was the data generated? Importantly, know the uncertainty in the data, not just the point estimate, and be prepared to feed that uncertainty into the calibration process downstream. Plot the data, including error bars for uncertainty.
Visualize early
Early and often, visually assess how well the predicted probabilities from the model align with observed frequency of events using diagnostic plots, also referred to as calibration curves. In a well-calibrated model, if a model predicts that an event has a 30% chance of occurring, this should align with the real-world observation of that event. There are a few different types of diagnostic plots, but generally they bin events by probability and graph predicted values against observed values to highlight deviations from an idea diagonal line.
For example, if the curve is above the diagonal, the model overestimates probabilities; if the curve is below the diagonal, the model underestimates probabilities. For more information see the tidyverse blog on calibration.
Examples
The following examples illustrate this concept further:
Develop a likelihood function
Use the diagnostic plots to develop a likelihood function. Think carefully about what data you'd like the model to recapitulate. While it's sometimes sufficient to directly use survey data, other calibrations will call for the development of summary statistics. Yet other calibrations may benefit from derived statistics specifically designed to encourage the model to bend in the desired way. These can be developed over time as you learn what is working or not in given a calibration.
For more information on the mathematics of developing likelihood functions, see pseudo-likelihood functions.
Examples
The following examples illustrate this concept further:
Check coverage
Before formally calibrating the model, simulate the model at a number of random parameter sets and evaluate if the data is "covered." For example, do the high and low outputs from the model capture the highs and lows seen in the data?
Handle edge cases
Watch out for edge cases. Simulations may burn out, resulting in zero cases or other such artifacts. Ensure that these edge cases are handled appropriately at an early stage or you'll end up wasting time later. In particular, look for parameter combinations within the search space that may cause problems. Reconsider burn-in and model seeding so as to mitigate issues.
Re-identify known parameters
The first calibration should not use real data, but rather data from a simulation with known parameters. Is the algorithm able to "re-identify" known parameters? Start by calibrating just two parameters. Not only is 2D relatively easy for most algorithm, but you can/should also visualize simulation outputs over the two input parameters. (Note that if the model is stochastic, it's unlikely to perfectly match the original parameters, especially if simulation replicates are not employed.)
Examples
The following examples illustrate this concept further:
- Fit a deterministic SIR model to synthetic data using optimization
- Fit a stochastic SIR model to synthetic data using Bayesian methods
Define parameters
Think very carefully about which parameters you will "open up" to the calibration. Note the distinction between calibration parameters and model parameters in the sense that each calibration parameter can modify one or possibly many model parameters. You'll achieve better results if calibration parameters act orthogonally on the model, or at least are non-redundant.
Examples
The following examples illustrate this concept further:
- Starsim's calibration class
- Fit a deterministic SIR model to synthetic data using optimization
- Fit a stochastic SIR model to synthetic data using Bayesian methods
Small calibration
Try a small calibration with the real data. By small, we mean to allow the algorithm to adjust only a few input parameters. It won't work. Well, everything should run by this point, but it's unlikely that the output from the model will look like the data. However, going through a small calibration process will illustrate what additional features are needed in the model to better capture reality.
Examples
The following examples illustrate this concept further:
Iterate
Run a calibration and look at the results. Likely, the first calibration doesn't fit the data well yet. In that case, check the following:
- Did any of the parameters hit up against their maximum or minimum limits? If so, consider the biological plausibility of an expanded range.
- Look at the parameter sensitivity plot and assess if there are any parameters that can be removed (fixed) for the next try. Think through which one or two more parameters to add next time.
- Think about the model—does it have the flexibility to hit the data? Make changes and start again if not.
- Revise feature weights and revisit feature engineering.
- Assess if edge cases are a problem and fix if they are.
Examples
The following examples illustrate this concept further: