T7 - Calibration#

We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.

Click here to open an interactive version of this notebook.

The Fit object#

The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:

[1]:

import covasim as cv
cv.options(jupyter=True, verbose=0)

pars = dict(
    pop_size  = 20_000,
    start_day = '2020-02-01',
    end_day   = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

Covasim 3.1.7 (2026-02-18) — © 2020-2026 by IDM

../_images/tutorials_tut_calibration_3_1.png

We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?

[2]:

fit = sim.compute_fit()
fit.summarize()

Mismatch values for:
#0. 'cum_deaths':    55.65217391304348
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 22.428498467824312

Total mismatch value:
78.08067238086778

So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)

What if we improve the fit? Does the mismatch reduce?

[3]:

sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim

# Rerun and compute fit
sim.run()
fit = sim.compute_fit()

# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

../_images/tutorials_tut_calibration_7_0.png

[4]:

fit.plot()
fit.summarize()

../_images/tutorials_tut_calibration_8_0.png

Mismatch values for:
#0. 'cum_deaths':    43.913043478260875
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 23.240551583248212

Total mismatch value:
67.1535950615091

As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:

\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)

where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).

Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.

Calibration approaches#

Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).

In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.

However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:

import covasim as cv
import numpy as np
from scipy import optimize

def objective(x, n_runs=10):
    print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
    pars = dict(
        pop_size       = 20_000,
        start_day      = '2020-02-01',
        end_day        = '2020-04-11',
        beta           = x[0],
        rel_death_prob = x[1],
        verbose        = 0,
    )
    sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
    msim = cv.MultiSim(sim)
    msim.run(n_runs=n_runs)
    mismatches = []
    for sim in msim.sims:
        fit = sim.compute_fit()
        mismatches.append(fit.mismatch)
    mismatch = np.mean(mismatches)
    return mismatch

guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization

This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.

What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.

Built-in calibration#

One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!

[5]:

'''
Example for running built-in calibration with Optuna
'''

import sciris as sc
import covasim as cv

# Create default simulation
pars = sc.objdict(
    pop_size       = 20_000,
    start_day      = '2020-02-01',
    end_day        = '2020-04-11',
    beta           = 0.015,
    rel_death_prob = 1.0,
    interventions  = cv.test_num(daily_tests='data'),
    verbose        = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')

# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
    beta           = [pars.beta, 0.005, 0.20],
    rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)

if __name__ == '__main__':

    # Run the calibration
    calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)

/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-covasim/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
[I 2026-03-18 15:13:19,223] A new study created in RDB with name: covasim_calibration

Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db

[I 2026-03-18 15:13:20,124] Trial 0 finished with value: 94.73397877159479 and parameters: {'beta': 0.014564708861155455, 'rel_death_prob': 1.9619468795653192}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:20,341] Trial 1 finished with value: 1983.457831860372 and parameters: {'beta': 0.10972696061921015, 'rel_death_prob': 2.6055564838791567}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:20,414] Trial 2 finished with value: 1437.0969711773328 and parameters: {'beta': 0.06993856816900582, 'rel_death_prob': 1.5832180218856104}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:20,496] Trial 3 finished with value: 1108.9399120664389 and parameters: {'beta': 0.1504765371008447, 'rel_death_prob': 0.8680468529846312}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:20,877] Trial 4 finished with value: 204.53668339476837 and parameters: {'beta': 0.01893945734024566, 'rel_death_prob': 2.0850422858819093}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:21,182] Trial 6 finished with value: 201.22918239552337 and parameters: {'beta': 0.01681507566451354, 'rel_death_prob': 1.5881226334492755}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:21,255] Trial 5 finished with value: 1119.9781276368965 and parameters: {'beta': 0.07210242503627395, 'rel_death_prob': 1.3029699428002826}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:21,362] Trial 7 finished with value: 928.7808100546254 and parameters: {'beta': 0.035567679462677984, 'rel_death_prob': 1.3957944635861974}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:21,798] Trial 8 finished with value: 1170.5768974552561 and parameters: {'beta': 0.09618246646126674, 'rel_death_prob': 1.1649005277832531}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:22,066] Trial 9 finished with value: 981.7167029355599 and parameters: {'beta': 0.05207881976896448, 'rel_death_prob': 1.2019405161803571}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:22,140] Trial 10 finished with value: 661.846604787494 and parameters: {'beta': 0.051283772800359416, 'rel_death_prob': 0.7414677481537308}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:22,277] Trial 11 finished with value: 916.1696718035262 and parameters: {'beta': 0.1019081172478049, 'rel_death_prob': 0.7607579745077668}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:22,777] Trial 12 finished with value: 1669.8428964782165 and parameters: {'beta': 0.16572730224000237, 'rel_death_prob': 1.5089702856285494}. Best is trial 0 with value: 94.73397877159479.
[I 2026-03-18 15:13:22,825] Trial 14 finished with value: 76.30312652662433 and parameters: {'beta': 0.01351685981370821, 'rel_death_prob': 2.052076916145788}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:22,887] Trial 15 finished with value: 137.92201447795 and parameters: {'beta': 0.0063526571450747745, 'rel_death_prob': 2.0317872442886618}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:23,058] Trial 13 finished with value: 2637.7967757694187 and parameters: {'beta': 0.19700840853987367, 'rel_death_prob': 2.176100586961106}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:23,352] Trial 16 finished with value: 141.3400985921748 and parameters: {'beta': 0.0053206066893717666, 'rel_death_prob': 2.047393594805501}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:23,849] Trial 18 finished with value: 2377.8422969312073 and parameters: {'beta': 0.13176238039683297, 'rel_death_prob': 2.620078580586686}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:23,861] Trial 17 finished with value: 2693.2382200115467 and parameters: {'beta': 0.1975184437946173, 'rel_death_prob': 2.708312916157864}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:23,959] Trial 19 finished with value: 1463.899609184172 and parameters: {'beta': 0.04188191013089401, 'rel_death_prob': 2.887938736530402}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:24,209] Trial 20 finished with value: 1493.3748945241373 and parameters: {'beta': 0.040115213894248394, 'rel_death_prob': 2.796354915780398}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:24,739] Trial 21 finished with value: 1215.3626149131767 and parameters: {'beta': 0.03699340597334576, 'rel_death_prob': 2.3706288950686263}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:24,755] Trial 22 finished with value: 1465.246591464227 and parameters: {'beta': 0.04278953802940831, 'rel_death_prob': 2.323691234194235}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:24,823] Trial 24 finished with value: 127.96698050361948 and parameters: {'beta': 0.008307010129055235, 'rel_death_prob': 1.8521644471276324}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:24,864] Trial 23 finished with value: 1842.8969223253541 and parameters: {'beta': 0.06724905672442683, 'rel_death_prob': 2.4077778550497912}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:25,310] Trial 25 finished with value: 141.3400985921748 and parameters: {'beta': 0.005324501889255914, 'rel_death_prob': 1.8472283076564657}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:25,406] Trial 26 finished with value: 126.36807745259136 and parameters: {'beta': 0.006584974817663832, 'rel_death_prob': 1.9319565869694317}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:25,657] Trial 27 finished with value: 332.7463694097793 and parameters: {'beta': 0.021113510089550955, 'rel_death_prob': 1.832707407409144}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:25,745] Trial 28 finished with value: 758.0505840031976 and parameters: {'beta': 0.02602143678536452, 'rel_death_prob': 1.77064107308744}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:26,109] Trial 29 finished with value: 339.70910867344674 and parameters: {'beta': 0.02183999929662985, 'rel_death_prob': 1.760967680683923}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:26,241] Trial 30 finished with value: 688.0415907980637 and parameters: {'beta': 0.025983112288989633, 'rel_death_prob': 1.7850877535242247}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:26,510] Trial 31 finished with value: 919.9970022649553 and parameters: {'beta': 0.026958760810465568, 'rel_death_prob': 2.2493821225550303}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:26,669] Trial 32 finished with value: 2024.1900564018297 and parameters: {'beta': 0.08992289501260824, 'rel_death_prob': 2.4616538567588404}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:27,033] Trial 33 finished with value: 1941.6988275525157 and parameters: {'beta': 0.08381478731262694, 'rel_death_prob': 2.433608585425075}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:27,142] Trial 34 finished with value: 1927.0109250788291 and parameters: {'beta': 0.08038018156303313, 'rel_death_prob': 2.219004099570859}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:27,405] Trial 35 finished with value: 1882.9827463694098 and parameters: {'beta': 0.05754493997515281, 'rel_death_prob': 2.461302977075129}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:27,540] Trial 36 finished with value: 1487.8269751743126 and parameters: {'beta': 0.05884211301336134, 'rel_death_prob': 1.9817395660983042}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:27,901] Trial 37 finished with value: 1593.037704845228 and parameters: {'beta': 0.05753244523955731, 'rel_death_prob': 1.963168034830775}. Best is trial 14 with value: 76.30312652662433.
[I 2026-03-18 15:13:28,072] Trial 39 finished with value: 74.10745214726651 and parameters: {'beta': 0.013018505245559316, 'rel_death_prob': 1.976434405995252}. Best is trial 39 with value: 74.10745214726651.
[I 2026-03-18 15:13:28,082] Trial 38 finished with value: 1829.7299817915355 and parameters: {'beta': 0.11555905710042101, 'rel_death_prob': 1.9423953776901302}. Best is trial 39 with value: 74.10745214726651.
[I 2026-03-18 15:13:28,228] Trial 40 finished with value: 58.105431451791986 and parameters: {'beta': 0.015880149400358964, 'rel_death_prob': 1.5963971001625215}. Best is trial 40 with value: 58.105431451791986.
[I 2026-03-18 15:13:28,772] Trial 43 finished with value: 56.643869076697605 and parameters: {'beta': 0.013054516353325482, 'rel_death_prob': 1.6316382687884121}. Best is trial 43 with value: 56.643869076697605.
[I 2026-03-18 15:13:28,812] Trial 42 finished with value: 44.07902917795444 and parameters: {'beta': 0.015143371678753452, 'rel_death_prob': 1.5319955458197168}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:28,826] Trial 41 finished with value: 1580.0866012346228 and parameters: {'beta': 0.11954269789859068, 'rel_death_prob': 1.5532622486000816}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:28,917] Trial 44 finished with value: 70.44044499711329 and parameters: {'beta': 0.014641942279870824, 'rel_death_prob': 1.6452357977831105}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:29,439] Trial 45 finished with value: 94.03794910512057 and parameters: {'beta': 0.01462535676205449, 'rel_death_prob': 1.0256901727698018}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:29,575] Trial 47 finished with value: 59.21081849269441 and parameters: {'beta': 0.015536636520003344, 'rel_death_prob': 1.6511791243146987}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:29,678] Trial 46 finished with value: 866.3557534307413 and parameters: {'beta': 0.03221579779563036, 'rel_death_prob': 1.4002906855057629}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:29,797] Trial 48 finished with value: 671.539392459031 and parameters: {'beta': 0.03309167803338703, 'rel_death_prob': 1.01727080284821}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:30,298] Trial 49 finished with value: 812.9242350224274 and parameters: {'beta': 0.030470947709119494, 'rel_death_prob': 1.6539458754140934}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:30,450] Trial 50 finished with value: 977.8334147532976 and parameters: {'beta': 0.03193605359591435, 'rel_death_prob': 1.4038450988808637}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:30,538] Trial 51 finished with value: 1201.1501310121241 and parameters: {'beta': 0.04928242729829298, 'rel_death_prob': 1.6608419876262759}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:30,681] Trial 52 finished with value: 1239.7425278678334 and parameters: {'beta': 0.04817457605048994, 'rel_death_prob': 1.6538756373531418}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:31,160] Trial 53 finished with value: 1046.5456099835678 and parameters: {'beta': 0.04593886000367619, 'rel_death_prob': 1.4249053435910584}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:31,170] Trial 54 finished with value: 71.25149886752232 and parameters: {'beta': 0.015756838583285283, 'rel_death_prob': 1.6818204669802566}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:31,255] Trial 55 finished with value: 72.88593063019053 and parameters: {'beta': 0.015968586373210775, 'rel_death_prob': 1.2726307464103126}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:31,355] Trial 56 finished with value: 73.37111959852555 and parameters: {'beta': 0.015725932336276713, 'rel_death_prob': 1.4612475016627857}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:31,956] Trial 58 finished with value: 181.32910689701114 and parameters: {'beta': 0.018207785210057094, 'rel_death_prob': 0.5176900980731369}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:31,971] Trial 57 finished with value: 210.50073277967758 and parameters: {'beta': 0.018475150074304687, 'rel_death_prob': 1.2910963124587131}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:32,072] Trial 59 finished with value: 509.5043744726206 and parameters: {'beta': 0.02277222135603421, 'rel_death_prob': 1.4718369722277145}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:32,159] Trial 60 finished with value: 323.58218235111246 and parameters: {'beta': 0.02046393652101215, 'rel_death_prob': 1.5918195744755606}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:32,833] Trial 61 finished with value: 1102.012035351068 and parameters: {'beta': 0.03718629485949867, 'rel_death_prob': 1.5893379045101221}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:32,877] Trial 62 finished with value: 1137.638895057068 and parameters: {'beta': 0.03835931567701475, 'rel_death_prob': 1.6132793234558178}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:33,067] Trial 63 finished with value: 1906.5953501798642 and parameters: {'beta': 0.17459672996184958, 'rel_death_prob': 1.6270908520764995}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:33,141] Trial 64 finished with value: 1568.425856019896 and parameters: {'beta': 0.16064113411892467, 'rel_death_prob': 1.2764751485274402}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:33,483] Trial 65 finished with value: 109.6058533552427 and parameters: {'beta': 0.012578794298848827, 'rel_death_prob': 1.283580204734527}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:33,505] Trial 66 finished with value: 124.44430874450416 and parameters: {'beta': 0.01091870127984634, 'rel_death_prob': 1.300546080493915}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:33,707] Trial 67 finished with value: 79.99666918328373 and parameters: {'beta': 0.011626412714169945, 'rel_death_prob': 1.2525734118239416}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:33,780] Trial 68 finished with value: 125.96482657547631 and parameters: {'beta': 0.011219450731892573, 'rel_death_prob': 1.2353505758255043}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:34,109] Trial 69 finished with value: 141.3400985921748 and parameters: {'beta': 0.00516022334552064, 'rel_death_prob': 1.101481333839781}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:34,359] Trial 70 finished with value: 713.1445130345961 and parameters: {'beta': 0.029295912158434315, 'rel_death_prob': 1.1824810875470564}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:34,556] Trial 71 finished with value: 739.7606253053249 and parameters: {'beta': 0.02652137784080408, 'rel_death_prob': 1.7006295480711475}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:34,631] Trial 72 finished with value: 649.6715814717768 and parameters: {'beta': 0.026915091393502108, 'rel_death_prob': 1.1221116213061073}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:34,954] Trial 73 finished with value: 654.4891637429498 and parameters: {'beta': 0.02616049171538599, 'rel_death_prob': 1.512401469496969}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:35,203] Trial 74 finished with value: 584.0295110361061 and parameters: {'beta': 0.024061310496011172, 'rel_death_prob': 1.4953654298729908}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:35,308] Trial 75 finished with value: 132.57716392059334 and parameters: {'beta': 0.0167712667986251, 'rel_death_prob': 1.5093648219176217}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:35,384] Trial 76 finished with value: 195.52904472176579 and parameters: {'beta': 0.018981202792229323, 'rel_death_prob': 1.5044239832444326}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:35,661] Trial 77 finished with value: 95.68192920904205 and parameters: {'beta': 0.0160483073380202, 'rel_death_prob': 1.4695262516374243}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:35,986] Trial 78 finished with value: 301.06386285917307 and parameters: {'beta': 0.01844166857437965, 'rel_death_prob': 1.364642491027387}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,035] Trial 79 finished with value: 101.56181995825375 and parameters: {'beta': 0.016200781491183872, 'rel_death_prob': 1.3499151893110573}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,071] Trial 80 finished with value: 54.05282675311987 and parameters: {'beta': 0.015692212194509184, 'rel_death_prob': 1.7287049747192715}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,218] Trial 81 finished with value: 141.64908735621975 and parameters: {'beta': 0.005035529211532714, 'rel_death_prob': 1.7168476961124162}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,622] Trial 82 finished with value: 137.12672647333125 and parameters: {'beta': 0.008773400804712398, 'rel_death_prob': 1.7267641886683394}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,697] Trial 83 finished with value: 126.05353732735267 and parameters: {'beta': 0.008897664443712133, 'rel_death_prob': 1.7314249758952691}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,712] Trial 84 finished with value: 137.92201447795 and parameters: {'beta': 0.006472899658323518, 'rel_death_prob': 1.7252501111814509}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:36,840] Trial 85 finished with value: 127.14537904694231 and parameters: {'beta': 0.009759792341994304, 'rel_death_prob': 1.8684241614693309}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:37,573] Trial 87 finished with value: 1215.2813429853 and parameters: {'beta': 0.04341439408747246, 'rel_death_prob': 1.8527734032880867}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:37,589] Trial 86 finished with value: 1841.2204112448371 and parameters: {'beta': 0.1363915986349846, 'rel_death_prob': 1.8005407117604992}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:37,618] Trial 88 finished with value: 1081.6879468845761 and parameters: {'beta': 0.03384627488481397, 'rel_death_prob': 1.7968317690870157}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:37,723] Trial 89 finished with value: 992.3320158102766 and parameters: {'beta': 0.03469964112644766, 'rel_death_prob': 1.7946242409837798}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:38,410] Trial 90 finished with value: 549.5079273437847 and parameters: {'beta': 0.0239691750906591, 'rel_death_prob': 1.4213835175192608}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:38,424] Trial 91 finished with value: 319.04827463694096 and parameters: {'beta': 0.021123143773265005, 'rel_death_prob': 2.124115430645191}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:38,459] Trial 92 finished with value: 290.8741173335702 and parameters: {'beta': 0.0207212831643706, 'rel_death_prob': 1.5590909342739259}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:38,540] Trial 93 finished with value: 605.582226762002 and parameters: {'beta': 0.022256151935122113, 'rel_death_prob': 1.559694400529857}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:39,086] Trial 95 finished with value: 87.4253897055558 and parameters: {'beta': 0.01421709847901478, 'rel_death_prob': 1.906909406447167}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:39,104] Trial 94 finished with value: 82.65754763067905 and parameters: {'beta': 0.014299415184239458, 'rel_death_prob': 1.9223092067914551}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:39,164] Trial 96 finished with value: 87.4253897055558 and parameters: {'beta': 0.014206941644777558, 'rel_death_prob': 2.0189492099169843}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:39,220] Trial 97 finished with value: 99.35892880934406 and parameters: {'beta': 0.014770194013295185, 'rel_death_prob': 1.687196118849874}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:39,689] Trial 98 finished with value: 1140.3758271528177 and parameters: {'beta': 0.030960551216256297, 'rel_death_prob': 2.0104800750764684}. Best is trial 42 with value: 44.07902917795444.
[I 2026-03-18 15:13:39,715] Trial 99 finished with value: 827.0269129990673 and parameters: {'beta': 0.028563253362186364, 'rel_death_prob': 1.609610530989607}. Best is trial 42 with value: 44.07902917795444.

Making results structure...
Processed 100 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 100 total trials completed in 21.4 s.

Initial parameter values:
#0. 'beta':           0.015
#1. 'rel_death_prob': 1.0

Best parameter values:
#0. 'beta':           0.015143371678753452
#1. 'rel_death_prob': 1.5319955458197168

Mismatch before calibration: 101.465
Mismatch after calibration:  44.079
Percent improvement:         56.6%

So it improved the fit (see above), but let’s visualize this as a plot:

[6]:

# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

../_images/tutorials_tut_calibration_16_0.png

../_images/tutorials_tut_calibration_16_1.png

Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.

Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.

[7]:

pars = dict(
    verbose = 0,
    start_day = '2020-02-05',
    pop_size = 1000,
    pop_scale = 4,
    interventions = cv.test_prob(symp_prob=0.01),
)

sim = cv.Sim(pars, datafile='example_data.csv')

calib_pars = dict(
    beta      = [0.013, 0.005, 0.020],
    test_prob = [0.01, 0.00, 0.30]
)

def set_test_prob(sim, calib_pars):
    tp = sim.get_intervention(cv.test_prob)
    tp.symp_prob = calib_pars['test_prob']
    return sim

calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])

Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db

[I 2026-03-18 15:13:42,029] A new study created in RDB with name: covasim_calibration
[I 2026-03-18 15:13:42,400] Trial 0 finished with value: 130.22406061882526 and parameters: {'beta': 0.016964602823069788, 'test_prob': 0.2625667798735441}. Best is trial 0 with value: 130.22406061882526.
[I 2026-03-18 15:13:42,469] Trial 2 finished with value: 80.27200918010954 and parameters: {'beta': 0.018934783208763767, 'test_prob': 0.19704636916181956}. Best is trial 2 with value: 80.27200918010954.
[I 2026-03-18 15:13:42,498] Trial 3 finished with value: 137.31188469494998 and parameters: {'beta': 0.006806579810162991, 'test_prob': 0.04660678261447043}. Best is trial 2 with value: 80.27200918010954.
[I 2026-03-18 15:13:42,500] Trial 1 finished with value: 83.77798579221914 and parameters: {'beta': 0.017706835884391255, 'test_prob': 0.15627616615705}. Best is trial 2 with value: 80.27200918010954.
[I 2026-03-18 15:13:42,602] Trial 4 finished with value: 126.23535828030236 and parameters: {'beta': 0.011046965091220915, 'test_prob': 0.2989288207683784}. Best is trial 2 with value: 80.27200918010954.
[I 2026-03-18 15:13:42,709] Trial 5 finished with value: 76.94317360787412 and parameters: {'beta': 0.013910906469519493, 'test_prob': 0.17072913121747751}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:42,733] Trial 7 finished with value: 139.82067160452215 and parameters: {'beta': 0.008471701328710654, 'test_prob': 0.26873472411431354}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:42,803] Trial 6 finished with value: 137.18301719264048 and parameters: {'beta': 0.01011251295077889, 'test_prob': 0.2673995546175842}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:42,813] Trial 8 finished with value: 100.61757124438714 and parameters: {'beta': 0.019671666523129957, 'test_prob': 0.2739954296325468}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:42,885] Trial 9 finished with value: 143.53078787306796 and parameters: {'beta': 0.012343793137619043, 'test_prob': 0.2446040379552174}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:42,978] Trial 10 finished with value: 141.03296697565293 and parameters: {'beta': 0.010353409453259799, 'test_prob': 0.20194373477321345}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,050] Trial 12 finished with value: 140.88608372582948 and parameters: {'beta': 0.01140294442693653, 'test_prob': 0.12887389468978414}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,066] Trial 11 finished with value: 135.68584942443573 and parameters: {'beta': 0.005753809430713972, 'test_prob': 0.07754904663251269}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,174] Trial 13 finished with value: 128.10180019457465 and parameters: {'beta': 0.01429664526415787, 'test_prob': 0.068011472136326}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,186] Trial 14 finished with value: 84.37442791674688 and parameters: {'beta': 0.015573901126659183, 'test_prob': 0.10262220253133825}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,278] Trial 15 finished with value: 106.85548055619988 and parameters: {'beta': 0.01469746988203437, 'test_prob': 0.18357409259382101}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,319] Trial 16 finished with value: 86.45471051698578 and parameters: {'beta': 0.014864897271744318, 'test_prob': 0.1853609904787152}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,408] Trial 17 finished with value: 127.84784708879184 and parameters: {'beta': 0.014273418388705378, 'test_prob': 0.18708969729930805}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,425] Trial 18 finished with value: 96.13870073210298 and parameters: {'beta': 0.01994937411975063, 'test_prob': 0.18683651206603133}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,520] Trial 19 finished with value: 105.6492008826936 and parameters: {'beta': 0.018853043942303645, 'test_prob': 0.2003445045809148}. Best is trial 5 with value: 76.94317360787412.
[I 2026-03-18 15:13:43,545] Trial 20 finished with value: 67.3073423732054 and parameters: {'beta': 0.019775447242669088, 'test_prob': 0.22311432949264465}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:43,649] Trial 21 finished with value: 95.77178383643364 and parameters: {'beta': 0.017910195274414238, 'test_prob': 0.0019660678317668656}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:43,674] Trial 22 finished with value: 73.19562460507089 and parameters: {'beta': 0.017811897596010758, 'test_prob': 0.0024011327527656834}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:43,773] Trial 23 finished with value: 89.2124536551535 and parameters: {'beta': 0.017398123605754393, 'test_prob': 0.2274712239717557}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:43,785] Trial 24 finished with value: 97.99263300639785 and parameters: {'beta': 0.017616339227306068, 'test_prob': 0.22852575518248874}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:43,885] Trial 25 finished with value: 95.92211889092133 and parameters: {'beta': 0.016250380293903452, 'test_prob': 0.2303031011803941}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:43,902] Trial 26 finished with value: 70.96392629829936 and parameters: {'beta': 0.016660835985995187, 'test_prob': 0.14798070357790102}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,003] Trial 27 finished with value: 97.19693949328894 and parameters: {'beta': 0.01593668228803637, 'test_prob': 0.15085513557133481}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,016] Trial 28 finished with value: 72.61049622848405 and parameters: {'beta': 0.01616045498483161, 'test_prob': 0.146704179027916}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,115] Trial 29 finished with value: 92.88798483832704 and parameters: {'beta': 0.018866016260498598, 'test_prob': 0.002635118523213308}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,138] Trial 30 finished with value: 135.7446229910958 and parameters: {'beta': 0.016170276608431008, 'test_prob': 0.009322796334961025}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,237] Trial 31 finished with value: 114.63414350075978 and parameters: {'beta': 0.013293067444460837, 'test_prob': 0.12174320230770624}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,254] Trial 32 finished with value: 133.887139068626 and parameters: {'beta': 0.013159926370323507, 'test_prob': 0.12218405478882427}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,355] Trial 33 finished with value: 79.22550330868899 and parameters: {'beta': 0.016924732457680714, 'test_prob': 0.12218039340917798}. Best is trial 20 with value: 67.3073423732054.
[I 2026-03-18 15:13:44,374] Trial 34 finished with value: 65.02738712806881 and parameters: {'beta': 0.01823481322799308, 'test_prob': 0.11966219350974679}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,494] Trial 35 finished with value: 103.41645978361291 and parameters: {'beta': 0.01828513258509911, 'test_prob': 0.0938211000956256}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,496] Trial 36 finished with value: 123.52145198050647 and parameters: {'beta': 0.016939964359991903, 'test_prob': 0.03394187467239003}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,579] Trial 37 finished with value: 70.3806516403724 and parameters: {'beta': 0.01844982295958193, 'test_prob': 0.14121461580255612}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,606] Trial 38 finished with value: 69.56362691027606 and parameters: {'beta': 0.018503597504761028, 'test_prob': 0.1417653930757765}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,709] Trial 39 finished with value: 107.93469880318992 and parameters: {'beta': 0.01915173220100154, 'test_prob': 0.16261660411795437}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,753] Trial 40 finished with value: 111.99792641667091 and parameters: {'beta': 0.01950610911899419, 'test_prob': 0.16465760355646789}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,816] Trial 41 finished with value: 111.32197547561232 and parameters: {'beta': 0.019278940361099037, 'test_prob': 0.16745927375661507}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,852] Trial 42 finished with value: 114.44370668394237 and parameters: {'beta': 0.019443380182576405, 'test_prob': 0.1629809360671864}. Best is trial 34 with value: 65.02738712806881.
[I 2026-03-18 15:13:44,957] Trial 43 finished with value: 59.45046568540341 and parameters: {'beta': 0.01839349239809394, 'test_prob': 0.10331325573319051}. Best is trial 43 with value: 59.45046568540341.
[I 2026-03-18 15:13:45,020] Trial 44 finished with value: 57.479385777475656 and parameters: {'beta': 0.018210622572550406, 'test_prob': 0.1387032072612841}. Best is trial 44 with value: 57.479385777475656.
[I 2026-03-18 15:13:45,068] Trial 45 finished with value: 92.49007055955276 and parameters: {'beta': 0.018686081849053234, 'test_prob': 0.140055632993356}. Best is trial 44 with value: 57.479385777475656.
[I 2026-03-18 15:13:45,108] Trial 46 finished with value: 57.07065283230176 and parameters: {'beta': 0.018263563481947926, 'test_prob': 0.13937005086664417}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,170] Trial 47 finished with value: 86.24072016955152 and parameters: {'beta': 0.018646405373179433, 'test_prob': 0.09943968971885422}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,263] Trial 48 finished with value: 91.17315220190089 and parameters: {'beta': 0.018363842877273946, 'test_prob': 0.102074722406656}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,300] Trial 49 finished with value: 138.71643803355636 and parameters: {'beta': 0.009119488886841578, 'test_prob': 0.10247834444418924}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,358] Trial 50 finished with value: 127.82799209471474 and parameters: {'beta': 0.007193588577183644, 'test_prob': 0.08076242023285582}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,416] Trial 51 finished with value: 104.45976950861721 and parameters: {'beta': 0.01748133513571123, 'test_prob': 0.07910181240481093}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,479] Trial 52 finished with value: 145.36642838530503 and parameters: {'beta': 0.007912769335772368, 'test_prob': 0.07023210785308967}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,548] Trial 53 finished with value: 78.71211932259865 and parameters: {'beta': 0.015293339811727752, 'test_prob': 0.2951627869919091}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,609] Trial 54 finished with value: 107.94107986160546 and parameters: {'beta': 0.019972922931379873, 'test_prob': 0.13422422951841814}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,664] Trial 55 finished with value: 115.96235169719375 and parameters: {'beta': 0.015349163884786549, 'test_prob': 0.13347888170390895}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,706] Trial 56 finished with value: 88.43063449522023 and parameters: {'beta': 0.01970887027000323, 'test_prob': 0.11177152683870989}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,778] Trial 57 finished with value: 59.82241847827876 and parameters: {'beta': 0.019648474190118618, 'test_prob': 0.12996701058168672}. Best is trial 46 with value: 57.07065283230176.
[I 2026-03-18 15:13:45,830] Trial 58 finished with value: 56.882524341186496 and parameters: {'beta': 0.01817389197681643, 'test_prob': 0.11870932813939444}. Best is trial 58 with value: 56.882524341186496.
[I 2026-03-18 15:13:45,871] Trial 59 finished with value: 67.84198000297025 and parameters: {'beta': 0.017912624754545706, 'test_prob': 0.11231987741472085}. Best is trial 58 with value: 56.882524341186496.

Making results structure...
Processed 60 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 60 total trials completed in 4.3 s.

Initial parameter values:
#0. 'beta':      0.013
#1. 'test_prob': 0.01

Best parameter values:
#0. 'beta':      0.01817389197681643
#1. 'test_prob': 0.11870932813939444

Mismatch before calibration: 141.953
Mismatch after calibration:  56.8825
Percent improvement:         59.9%

../_images/tutorials_tut_calibration_18_3.png

../_images/tutorials_tut_calibration_18_4.png