Introduction to model calibration

In health and disease modeling, we seek to inform decision making through scenario analyses, projections, and other insights that are accurate and robust to uncertainties. To achieve this aim, we leverage models that include dynamics relevant to the decisions under consideration and produce outputs that sufficiently recapitulate the observed real-world data. Calibration is the process of building and iteratively refining models to ensure they meet these criteria.

The concept is simple, but calibration is often the most challenging aspect of the modeling process. In the narrow view of calibration, we adjust input parameters of an established model until the outputs align closely with observed data. However, model-based guidance often benefits from a broader view of calibration that touches on all aspects of modeling:

• Model design in light key questions and diverse scientific hypotheses
• Data assimilation and feature engineering
• Parameter and prior selection
• Output modeling and pseudo-likelihood development
• Model diagnostics and visualization

Small exploratory calibrations can be done by hand on a personal computer. However, most analyses require iterative workflows that run sophisticated algorithms on scaled computing platforms. While more resource intensive, these workflows are essential for robustly quantifying uncertainty surrounding key decisions.

The mathematics of model calibration rests upon core principles from statistics. This theory provides a framework on which we can cast the calibration problem as optimization, posterior sampling, model selection / ensemble building, or other formulations. The appropriate computational methodology will depend on the analysis endpoints, data availability, operational context, and resource budget of the overall effort.

Model calibration is an expansive topic that touches on all aspects of health and disease modeling. Therefore, we have organized content into two main groups:

• Process best practices cover aspects of the calibration techniques themselves.
• Operational best practices address best practices for your workflow.

The best practices described in each group take up-front time to implement. However, we have found that these investments are time-saving in the long run and lead to more robust and reliable results. While there is no one "right" way to calibrate a model, this guidance is built on decades of experience across dozens of projects.

As you review the content, you'll notice that we've included links to real-world examples of model calibration to illustrate these best practices. We are working to make all of the examples public, but please note that some examples may still be private at this time.