Model calibration using optimization
Model calibration seeks to contextualize a model to a specific setting using data. The math and methods described above focus on obtaining samples from the posterior distribution of the joint distribution of parameters and latent states. However, an alternative approach is to use numerical optimization to find the single best-fit parameter set according to some loss function, \(f\),
The appeal of optimization-based approaches to calibration is that they are often simpler to implement and can be computationally cheaper than full posterior sampling. However, optimization yields only a point estimate of the best parameters, without a direct quantification of uncertainty.
Most often, we choose to maximize the (pseudo)-likelihood function, \(f(y^{\text{obs}}, \tilde y(\theta)) = \mathcal L(\theta)\), but a common alternatives include absolute and squared error (result is the same as a normal likelihood).
Limitations of optimization-based calibration
Optimization-based approaches to calibration have several limitations. First, they typically provide only a point estimate of the best parameters, without a direct quantification of uncertainty. Other methods are required to assess local uncertainty around the optimum, such as the Fisher information or bootstrap. However, these methods require additional simulations to be run and often rely on assumptions that may not hold in practice.
Optimization-based approaches seek the "best" parameters without jointly estimating the latent state trajectory. Simulations using the best parameters do not necessarily yield latent state trajectories that are consistent with the observed data. Therefore, optimization-based calibration is particularly ill-suited to models with intrinsic stochasticity.
Further, this class of methods finds only one solution. While optimization users will often keep the best \(K\) parameters and latent trajectories found during the search, these are not samples from the posterior distribution and do not have any statistical meaning.
Finally, the loss function used in optimization may not have a well-defined maximum. For example, if the likelihood function is flat or multi-modal, optimization will arbitrarily select one point. As a diagnostic, you should visualize the loss surface near the optimum using techniques like gradient checks and profiles to assess if the solution is well-defined. To be clear, the issue is that other parameters may equally well explain the data, yet produce different answers to the underlying modeling question.
Note
An optimization-based approach to the calibration problem should be viewed as exploratory. Optimization results should not be used to make statistical inferences about the model parameters and do not provide a basis for quantifying uncertainty in model outputs without additional analysis.
Numerical methods
We have had good success with a class of methods known as Bayesian optimization, using Optuna specifically. Bayesian optimization is a global optimization technique (as opposed to a local gradient method) that typically places each new simulation at the point of maximal expected improvement. Numerous algorithm tweaks are often required to consistently produce reliable results.
Posterior sampling
Instead, we recommend using methods that seek to find a collection of joint parameter and latent state trajectory samples from the posterior distribution, following the Bayesian formulation described above.
While a full Bayesian analysis is typical for statistical and deterministic epidemiology models using techniques like Markov Chain Monte Carlo (MCMC), it is less common in agent-based models due to their high-dimensional input spaces, stochasticity, and computational intensity. Nonetheless, posterior sampling is the "gold standard" for model calibration, providing a robust quantification of uncertainty.
Traditional single-chain MCMC is often inefficient for stochastic models such as agent-based models. Each likelihood evaluation may require multiple expensive simulator runs and the many serial evaluations needed for burn-in and mixing can make computation prohibitive. While Metropolis–Hastings acceptance rules still apply if you can compute an unbiased likelihood estimate, high-variance estimates lead to poor mixing, often requiring advanced methods.
A promising alternative for capturing parametric uncertainty is History Matching 1. This algorithm starts with a set of parameters and iteratively rules out parameter regions that are implausible given the data. On each wave (iteration), one or more outputs are emulated. We describe emulation, as used in history matching and Bayesian optimization, in the next section.
References
-
Vernon, I., Goldstein, M., & Bower, R. G. (2010). Galaxy formation: a Bayesian uncertainty analysis. Bayesian Analysis, 5(4), 619–670. https://doi.org/10.1214/10-BA524 ↩