T7 - Calibration

We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.

Click here to open an interactive version of this notebook.

The Fit object

The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:

[1]:
import covasim as cv
cv.options(jupyter=True, verbose=0)

pars = dict(
    pop_size  = 20_000,
    start_day = '2020-02-01',
    end_day   = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
Covasim 3.1.4 (2022-10-22) — © 2020-2022 by IDM
../_images/tutorials_tut_calibration_3_1.svg

We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?

[2]:
fit = sim.compute_fit()
fit.summarize()
Mismatch values for:
#0. 'cum_deaths':    55.65217391304348
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 22.428498467824312

Total mismatch value:
78.08067238086778

So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)

What if we improve the fit? Does the mismatch reduce?

[3]:
sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim

# Rerun and compute fit
sim.run()
fit = sim.compute_fit()

# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
../_images/tutorials_tut_calibration_7_0.svg
[4]:
fit.plot()
fit.summarize()
../_images/tutorials_tut_calibration_8_0.svg
Mismatch values for:
#0. 'cum_deaths':    43.913043478260875
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 23.240551583248212

Total mismatch value:
67.1535950615091

As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:

\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)

where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).

Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.

Calibration approaches

Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).

In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.

However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:

import covasim as cv
import numpy as np
from scipy import optimize

def objective(x, n_runs=10):
    print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
    pars = dict(
        pop_size       = 20_000,
        start_day      = '2020-02-01',
        end_day        = '2020-04-11',
        beta           = x[0],
        rel_death_prob = x[1],
        verbose        = 0,
    )
    sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
    msim = cv.MultiSim(sim)
    msim.run(n_runs=n_runs)
    mismatches = []
    for sim in msim.sims:
        fit = sim.compute_fit()
        mismatches.append(fit.mismatch)
    mismatch = np.mean(mismatches)
    return mismatch

guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization

This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.

What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.

Built-in calibration

One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!

[5]:
'''
Example for running built-in calibration with Optuna
'''

import sciris as sc
import covasim as cv

# Create default simulation
pars = sc.objdict(
    pop_size       = 20_000,
    start_day      = '2020-02-01',
    end_day        = '2020-04-11',
    beta           = 0.015,
    rel_death_prob = 1.0,
    interventions  = cv.test_num(daily_tests='data'),
    verbose        = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')

# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
    beta           = [pars.beta, 0.005, 0.20],
    rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)

if __name__ == '__main__':

    # Run the calibration
    calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)
/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-covasim/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
[I 2022-10-28 22:18:44,721] A new study created in RDB with name: covasim_calibration
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db
[I 2022-10-28 22:18:45,614] Trial 1 finished with value: 130.64318070790955 and parameters: {'beta': 0.006983043771282648, 'rel_death_prob': 2.9393882924591748}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:46,051] Trial 0 finished with value: 2591.6336545721015 and parameters: {'beta': 0.16953093429129923, 'rel_death_prob': 2.8007800313524855}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:46,728] Trial 2 finished with value: 2150.9785939512367 and parameters: {'beta': 0.11583234588681068, 'rel_death_prob': 2.065346590630758}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:47,140] Trial 3 finished with value: 1822.7886707820758 and parameters: {'beta': 0.1006585761406435, 'rel_death_prob': 1.563666610549638}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:47,888] Trial 4 finished with value: 2626.379402229427 and parameters: {'beta': 0.19782813902446544, 'rel_death_prob': 2.4911925213917088}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:48,227] Trial 5 finished with value: 2222.9241239952034 and parameters: {'beta': 0.1074802789190168, 'rel_death_prob': 2.8097822515566038}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:48,972] Trial 6 finished with value: 2487.1943420526713 and parameters: {'beta': 0.10427967496384911, 'rel_death_prob': 2.752224236149676}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:49,207] Trial 7 finished with value: 1018.5640849136208 and parameters: {'beta': 0.028544676858374776, 'rel_death_prob': 2.1163889445164417}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:50,088] Trial 8 finished with value: 2204.303859306302 and parameters: {'beta': 0.13349768183650187, 'rel_death_prob': 2.8108025664653375}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:50,310] Trial 9 finished with value: 2224.190722565173 and parameters: {'beta': 0.11868670695970551, 'rel_death_prob': 2.2266178464118704}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:50,775] Trial 10 finished with value: 143.6877692410179 and parameters: {'beta': 0.008039211000186078, 'rel_death_prob': 0.6704940370144157}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:51,015] Trial 11 finished with value: 134.02329351156905 and parameters: {'beta': 0.009416475098713736, 'rel_death_prob': 0.7710877671229738}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:51,473] Trial 12 finished with value: 137.85406581693832 and parameters: {'beta': 0.009771496440515866, 'rel_death_prob': 0.6540412589310773}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:52,065] Trial 13 finished with value: 1037.2450814939823 and parameters: {'beta': 0.05923967603559517, 'rel_death_prob': 1.1528376095345738}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:52,494] Trial 14 finished with value: 1212.089643380557 and parameters: {'beta': 0.051177309990307535, 'rel_death_prob': 1.229032629619001}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:53,112] Trial 15 finished with value: 1231.2212994626284 and parameters: {'beta': 0.05885907840924091, 'rel_death_prob': 1.4114531779366786}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:53,518] Trial 16 finished with value: 1429.525469645157 and parameters: {'beta': 0.05636830217356345, 'rel_death_prob': 1.693798837617485}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:54,119] Trial 17 finished with value: 982.509881422925 and parameters: {'beta': 0.03566140454541737, 'rel_death_prob': 1.8384439162670414}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:54,514] Trial 18 finished with value: 727.3091442021583 and parameters: {'beta': 0.0360233326516214, 'rel_death_prob': 0.9488752568640858}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:55,173] Trial 19 finished with value: 1009.8625482968424 and parameters: {'beta': 0.07505265443290471, 'rel_death_prob': 0.9991082433396615}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:55,576] Trial 20 finished with value: 2118.9131545054847 and parameters: {'beta': 0.08092903555889097, 'rel_death_prob': 2.435005591555141}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:56,229] Trial 21 finished with value: 1836.9092241417598 and parameters: {'beta': 0.08417166089476902, 'rel_death_prob': 2.28674464413172}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:56,234] Trial 22 finished with value: 141.3400985921748 and parameters: {'beta': 0.005202820891857878, 'rel_death_prob': 0.6485803494926753}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:57,129] Trial 23 finished with value: 252.11962073100324 and parameters: {'beta': 0.019499650470633387, 'rel_death_prob': 0.8226451661572164}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:57,182] Trial 24 finished with value: 319.3239552338233 and parameters: {'beta': 0.02195737411521821, 'rel_death_prob': 0.5375799901776973}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:58,037] Trial 25 finished with value: 337.10607540969045 and parameters: {'beta': 0.021792064888023002, 'rel_death_prob': 1.3037108046166597}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:58,203] Trial 26 finished with value: 922.7294932717502 and parameters: {'beta': 0.03910860150299212, 'rel_death_prob': 1.1933843147944687}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:58,888] Trial 28 finished with value: 145.67837633787803 and parameters: {'beta': 0.005623478528278714, 'rel_death_prob': 0.8326263828687956}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:59,038] Trial 27 finished with value: 649.6152906692721 and parameters: {'beta': 0.04203285623199213, 'rel_death_prob': 0.8301120091062891}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:18:59,907] Trial 29 finished with value: 1339.5410134564995 and parameters: {'beta': 0.04537766246386025, 'rel_death_prob': 1.9036623464905627}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:19:00,143] Trial 30 finished with value: 2630.535151219079 and parameters: {'beta': 0.1463518343906412, 'rel_death_prob': 2.9803763810949135}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:19:00,831] Trial 32 finished with value: 144.1627437047564 and parameters: {'beta': 0.005973002200667007, 'rel_death_prob': 0.5035372860188309}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:19:01,027] Trial 31 finished with value: 2636.0530932184574 and parameters: {'beta': 0.1486469039022626, 'rel_death_prob': 2.9876298515727457}. Best is trial 1 with value: 130.64318070790955.
[I 2022-10-28 22:19:01,626] Trial 33 finished with value: 84.94726206865923 and parameters: {'beta': 0.016537046686742222, 'rel_death_prob': 0.6602481169290693}. Best is trial 33 with value: 84.94726206865923.
[I 2022-10-28 22:19:01,784] Trial 34 finished with value: 70.42445707687524 and parameters: {'beta': 0.015552251822198436, 'rel_death_prob': 0.6825482452960914}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:02,511] Trial 35 finished with value: 294.7949327175023 and parameters: {'beta': 0.019405705477716424, 'rel_death_prob': 0.9884975739110538}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:02,700] Trial 36 finished with value: 277.7173913043478 and parameters: {'beta': 0.02081727881786831, 'rel_death_prob': 1.0246439052795608}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:03,652] Trial 37 finished with value: 2101.0554247901587 and parameters: {'beta': 0.19178259788253554, 'rel_death_prob': 1.5092710701567724}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:03,763] Trial 38 finished with value: 1452.1160678598392 and parameters: {'beta': 0.07023313725311361, 'rel_death_prob': 1.4811661364896729}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:04,652] Trial 39 finished with value: 592.3941910556468 and parameters: {'beta': 0.03292507145918062, 'rel_death_prob': 0.7221898059294485}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:04,744] Trial 40 finished with value: 728.387551627659 and parameters: {'beta': 0.02997750151231, 'rel_death_prob': 1.6471702704454783}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:05,575] Trial 42 finished with value: 138.86829950703913 and parameters: {'beta': 0.017369214386095743, 'rel_death_prob': 0.6455994142720106}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:05,628] Trial 41 finished with value: 665.1975174312742 and parameters: {'beta': 0.027425751110174396, 'rel_death_prob': 1.672481242396666}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:06,297] Trial 43 finished with value: 102.78200914864324 and parameters: {'beta': 0.01416954701694556, 'rel_death_prob': 0.7767061043092843}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:06,345] Trial 44 finished with value: 113.728627259404 and parameters: {'beta': 0.012428254560215641, 'rel_death_prob': 0.7379480290662108}. Best is trial 34 with value: 70.42445707687524.
[I 2022-10-28 22:19:07,085] Trial 46 finished with value: 99.08280410356619 and parameters: {'beta': 0.014384107710811615, 'rel_death_prob': 1.1008457736853314}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:07,086] Trial 45 finished with value: 50.80761202646889 and parameters: {'beta': 0.015695934039206508, 'rel_death_prob': 2.6235715697012956}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:08,132] Trial 47 finished with value: 1644.2104854110228 and parameters: {'beta': 0.04837151569768784, 'rel_death_prob': 2.6082202102799257}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:08,133] Trial 48 finished with value: 803.6923213571968 and parameters: {'beta': 0.04564531341538072, 'rel_death_prob': 1.095442736152882}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:09,143] Trial 50 finished with value: 521.2332904028067 and parameters: {'beta': 0.02740917846735432, 'rel_death_prob': 0.9002323111910446}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:09,188] Trial 49 finished with value: 997.1044100013323 and parameters: {'beta': 0.09254919192432122, 'rel_death_prob': 0.9274509833092277}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:09,921] Trial 52 finished with value: 126.07063551982947 and parameters: {'beta': 0.01229685829927249, 'rel_death_prob': 0.5887354917965488}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:10,172] Trial 51 finished with value: 644.9916729582094 and parameters: {'beta': 0.06393060563549985, 'rel_death_prob': 0.5967989471387267}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:10,648] Trial 53 finished with value: 105.36350313096773 and parameters: {'beta': 0.014866677095636964, 'rel_death_prob': 0.7355582333596893}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:10,919] Trial 54 finished with value: 82.54274548119199 and parameters: {'beta': 0.013820654510484545, 'rel_death_prob': 0.7438410944563496}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:11,460] Trial 55 finished with value: 100.03186481325221 and parameters: {'beta': 0.016207505738600376, 'rel_death_prob': 2.0391327390703764}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:11,884] Trial 56 finished with value: 598.5558688990541 and parameters: {'beta': 0.027142241133179916, 'rel_death_prob': 1.0790591055388281}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:12,440] Trial 57 finished with value: 837.0030421459342 and parameters: {'beta': 0.027675322608406855, 'rel_death_prob': 1.9537380929875445}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:12,888] Trial 58 finished with value: 1190.8774481502865 and parameters: {'beta': 0.03705190336950942, 'rel_death_prob': 2.1025452813123544}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:13,445] Trial 59 finished with value: 1285.672380867789 and parameters: {'beta': 0.03847187947725959, 'rel_death_prob': 2.1127714529788273}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:13,963] Trial 60 finished with value: 2070.679375582893 and parameters: {'beta': 0.11213178768777862, 'rel_death_prob': 2.269599561786065}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:14,466] Trial 61 finished with value: 1462.7981080961051 and parameters: {'beta': 0.050545393837634026, 'rel_death_prob': 2.2736271142169215}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:14,760] Trial 62 finished with value: 107.08864413554203 and parameters: {'beta': 0.016041141284904317, 'rel_death_prob': 2.4188497719967748}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:15,212] Trial 63 finished with value: 71.24650264244792 and parameters: {'beta': 0.013574613342356807, 'rel_death_prob': 2.411685804947311}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:15,484] Trial 64 finished with value: 53.41941644091131 and parameters: {'beta': 0.011378842183718773, 'rel_death_prob': 2.6610456293152382}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:16,177] Trial 65 finished with value: 845.7976195763201 and parameters: {'beta': 0.02434564231127878, 'rel_death_prob': 2.693016976440561}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:16,448] Trial 66 finished with value: 765.9831460674158 and parameters: {'beta': 0.024098317701015767, 'rel_death_prob': 2.753130576765804}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:16,825] Trial 67 finished with value: 141.3400985921748 and parameters: {'beta': 0.005190369746917885, 'rel_death_prob': 2.879072577182772}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:17,146] Trial 68 finished with value: 120.87256295243594 and parameters: {'beta': 0.009413887868616636, 'rel_death_prob': 2.566216148541801}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:17,828] Trial 69 finished with value: 1260.1046986721144 and parameters: {'beta': 0.033514952518044726, 'rel_death_prob': 2.645025266738149}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:18,144] Trial 70 finished with value: 1076.6979393347249 and parameters: {'beta': 0.03535513023103694, 'rel_death_prob': 2.4494072843322727}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:18,857] Trial 71 finished with value: 1581.720810942843 and parameters: {'beta': 0.054419678107150996, 'rel_death_prob': 2.501091446045067}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:18,951] Trial 72 finished with value: 100.5783408091664 and parameters: {'beta': 0.016595539593629145, 'rel_death_prob': 2.507267892556035}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:19,731] Trial 73 finished with value: 395.76730914420216 and parameters: {'beta': 0.018434267997265, 'rel_death_prob': 2.864905620167187}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:20,054] Trial 74 finished with value: 2529.1332104632056 and parameters: {'beta': 0.12458256539359087, 'rel_death_prob': 2.8582001217042237}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:20,779] Trial 76 finished with value: 109.91717369098902 and parameters: {'beta': 0.010360108188623892, 'rel_death_prob': 2.361963792516703}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:20,832] Trial 75 finished with value: 2365.1155793400544 and parameters: {'beta': 0.12579306401602966, 'rel_death_prob': 2.7724923339945184}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:21,789] Trial 78 finished with value: 540.1001465559355 and parameters: {'beta': 0.022160022608458847, 'rel_death_prob': 2.1727374865217737}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:21,795] Trial 77 finished with value: 1274.6947861615668 and parameters: {'beta': 0.042099704159290235, 'rel_death_prob': 1.9929772854108105}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:22,511] Trial 79 finished with value: 98.3008393658125 and parameters: {'beta': 0.01062573565543777, 'rel_death_prob': 2.3316622853633624}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:22,790] Trial 80 finished with value: 1008.8736288137851 and parameters: {'beta': 0.03115243698810935, 'rel_death_prob': 1.8014785912125002}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:23,224] Trial 81 finished with value: 121.13281076519964 and parameters: {'beta': 0.010024583896390285, 'rel_death_prob': 2.3606972097933165}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:23,471] Trial 82 finished with value: 141.73857529866322 and parameters: {'beta': 0.008144335938441532, 'rel_death_prob': 2.179889898499294}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:23,974] Trial 83 finished with value: 69.46396056313009 and parameters: {'beta': 0.013452372335035405, 'rel_death_prob': 2.043792294323708}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:24,192] Trial 84 finished with value: 92.63989430208287 and parameters: {'beta': 0.014836250637259088, 'rel_death_prob': 2.6762257389013477}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:24,914] Trial 85 finished with value: 797.2883821112937 and parameters: {'beta': 0.022721162666426493, 'rel_death_prob': 2.347398150505968}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:25,135] Trial 86 finished with value: 653.3811120486744 and parameters: {'beta': 0.022605176444535833, 'rel_death_prob': 2.5678318102674984}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:25,554] Trial 87 finished with value: 146.63343251765332 and parameters: {'beta': 0.005563086567175822, 'rel_death_prob': 2.7100040023340393}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:25,815] Trial 88 finished with value: 137.92201447795 and parameters: {'beta': 0.006381823707952717, 'rel_death_prob': 2.665148528103587}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:26,274] Trial 89 finished with value: 103.11409157525426 and parameters: {'beta': 0.012193212116120034, 'rel_death_prob': 2.6457261165208994}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:26,531] Trial 90 finished with value: 105.0 and parameters: {'beta': 0.012326741231171811, 'rel_death_prob': 2.560123349706204}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:27,276] Trial 91 finished with value: 1123.6265932406625 and parameters: {'beta': 0.03051045188684206, 'rel_death_prob': 2.529569801771645}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:27,384] Trial 92 finished with value: 163.68210685260027 and parameters: {'beta': 0.018902374151180915, 'rel_death_prob': 1.260406601654101}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:28,133] Trial 93 finished with value: 167.18146289470178 and parameters: {'beta': 0.01816602600711862, 'rel_death_prob': 1.3167292420566024}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:28,538] Trial 94 finished with value: 963.9812141937202 and parameters: {'beta': 0.17707498596480753, 'rel_death_prob': 0.6880106501576794}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:29,157] Trial 95 finished with value: 555.166540835813 and parameters: {'beta': 0.02673769372230905, 'rel_death_prob': 0.8725582499071048}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:29,516] Trial 96 finished with value: 552.9625394146644 and parameters: {'beta': 0.02545462181704375, 'rel_death_prob': 0.8493125041622539}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:29,878] Trial 97 finished with value: 76.89678909268552 and parameters: {'beta': 0.01441868652931774, 'rel_death_prob': 2.3983675847071813}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:30,253] Trial 98 finished with value: 110.60909535017986 and parameters: {'beta': 0.014497522019296253, 'rel_death_prob': 0.5430266586998075}. Best is trial 45 with value: 50.80761202646889.
[I 2022-10-28 22:19:30,710] Trial 99 finished with value: 2174.8809788160056 and parameters: {'beta': 0.09863948140845563, 'rel_death_prob': 2.437829093545553}. Best is trial 45 with value: 50.80761202646889.
Making results structure...
Processed 100 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 100 total trials completed in 46.6 s.

Initial parameter values:
#0. 'beta':           0.015
#1. 'rel_death_prob': 1.0

Best parameter values:
#0. 'beta':           0.015695934039206508
#1. 'rel_death_prob': 2.6235715697012956

Mismatch before calibration: 101.465
Mismatch after calibration:  50.8076
Percent improvement:         49.9%

So it improved the fit (see above), but let’s visualize this as a plot:

[6]:
# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
../_images/tutorials_tut_calibration_16_0.svg
../_images/tutorials_tut_calibration_16_1.svg

Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.

Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.

[7]:
pars = dict(
    verbose = 0,
    start_day = '2020-02-05',
    pop_size = 1000,
    pop_scale = 4,
    interventions = cv.test_prob(symp_prob=0.01),
)

sim = cv.Sim(pars, datafile='example_data.csv')

calib_pars = dict(
    beta      = [0.013, 0.005, 0.020],
    test_prob = [0.01, 0.00, 0.30]
)

def set_test_prob(sim, calib_pars):
    tp = sim.get_intervention(cv.test_prob)
    tp.symp_prob = calib_pars['test_prob']
    return sim

calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db
[I 2022-10-28 22:19:34,137] A new study created in RDB with name: covasim_calibration
[I 2022-10-28 22:19:34,575] Trial 1 finished with value: 134.40304888697725 and parameters: {'beta': 0.010695450015723204, 'test_prob': 0.15453998912265335}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:34,577] Trial 0 finished with value: 74.42167286274092 and parameters: {'beta': 0.019925561821308328, 'test_prob': 0.24599259077545482}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:34,813] Trial 2 finished with value: 145.3038078348303 and parameters: {'beta': 0.005266612303049877, 'test_prob': 0.23724496589761193}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:34,855] Trial 3 finished with value: 143.35840099529358 and parameters: {'beta': 0.008509553219536545, 'test_prob': 0.05582804610504368}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,091] Trial 4 finished with value: 134.2520734311572 and parameters: {'beta': 0.014825702671847196, 'test_prob': 0.1790192907716824}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,165] Trial 5 finished with value: 95.16500551506641 and parameters: {'beta': 0.01861512246964075, 'test_prob': 0.012309756898688929}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,378] Trial 6 finished with value: 90.94272489486539 and parameters: {'beta': 0.012504273907287988, 'test_prob': 0.15115059748432078}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,459] Trial 7 finished with value: 100.31356633077579 and parameters: {'beta': 0.01998285108466925, 'test_prob': 0.10700005250008905}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,653] Trial 8 finished with value: 109.7769104360432 and parameters: {'beta': 0.012589405806623138, 'test_prob': 0.19590527184863923}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,719] Trial 9 finished with value: 119.55754533569353 and parameters: {'beta': 0.009092728424576564, 'test_prob': 0.2790369388155321}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:35,936] Trial 10 finished with value: 124.69186897413996 and parameters: {'beta': 0.015906976413003433, 'test_prob': 0.2847425559663614}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,008] Trial 11 finished with value: 114.20312414758952 and parameters: {'beta': 0.016736367241010547, 'test_prob': 0.28636337314738236}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,234] Trial 12 finished with value: 83.92427649988684 and parameters: {'beta': 0.016638273711864698, 'test_prob': 0.10865529802193033}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,294] Trial 13 finished with value: 78.9805581333065 and parameters: {'beta': 0.013547702095306828, 'test_prob': 0.12474507997393251}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,519] Trial 14 finished with value: 121.59879009523587 and parameters: {'beta': 0.0180179069805992, 'test_prob': 0.20898170914986355}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,588] Trial 15 finished with value: 118.56884543231395 and parameters: {'beta': 0.013812083312311034, 'test_prob': 0.2263293949279684}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,802] Trial 16 finished with value: 118.05107417160795 and parameters: {'beta': 0.014060786185565607, 'test_prob': 0.10497569163213644}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:36,827] Trial 17 finished with value: 145.5305920300728 and parameters: {'beta': 0.005857846768278695, 'test_prob': 0.10158894884928599}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,067] Trial 18 finished with value: 143.9619072002062 and parameters: {'beta': 0.00996287571148674, 'test_prob': 0.2441632760055892}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,107] Trial 19 finished with value: 137.70850228665526 and parameters: {'beta': 0.010586070837351707, 'test_prob': 0.24999134831497014}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,364] Trial 20 finished with value: 92.90854193720494 and parameters: {'beta': 0.018633035011743505, 'test_prob': 0.06764918467756349}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,386] Trial 21 finished with value: 143.62691459878806 and parameters: {'beta': 0.007587893557282405, 'test_prob': 0.05839028810030378}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,639] Trial 22 finished with value: 94.27491291888515 and parameters: {'beta': 0.016791235899686844, 'test_prob': 0.1291258108374623}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,691] Trial 23 finished with value: 93.93055512305065 and parameters: {'beta': 0.01684028241750003, 'test_prob': 0.12924630482695895}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,920] Trial 24 finished with value: 84.73593537476047 and parameters: {'beta': 0.01993055244018202, 'test_prob': 0.16874898006370503}. Best is trial 0 with value: 74.42167286274092.
[I 2022-10-28 22:19:37,998] Trial 25 finished with value: 49.57462390802959 and parameters: {'beta': 0.019701393540332103, 'test_prob': 0.1748859516818704}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:38,206] Trial 26 finished with value: 97.37108528239338 and parameters: {'beta': 0.014779980829225681, 'test_prob': 0.0837950411399628}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:38,276] Trial 27 finished with value: 127.16140190123124 and parameters: {'beta': 0.018358914221267814, 'test_prob': 0.18254515790110254}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:38,487] Trial 28 finished with value: 115.26883399608838 and parameters: {'beta': 0.01797274701881114, 'test_prob': 0.20346584323175304}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:38,567] Trial 29 finished with value: 76.01895201863329 and parameters: {'beta': 0.01927888522777421, 'test_prob': 0.2126983014924866}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:38,734] Trial 30 finished with value: 142.5488818208121 and parameters: {'beta': 0.011820534697289185, 'test_prob': 0.1352699676125347}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:38,862] Trial 31 finished with value: 88.8628625649503 and parameters: {'beta': 0.019373844097094316, 'test_prob': 0.2593561011986371}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,020] Trial 32 finished with value: 89.39196042667271 and parameters: {'beta': 0.01937113271317651, 'test_prob': 0.26732208857387735}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,138] Trial 33 finished with value: 114.57596914094587 and parameters: {'beta': 0.019314458200151072, 'test_prob': 0.15963249933378967}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,307] Trial 34 finished with value: 84.02741627432957 and parameters: {'beta': 0.0175321226253379, 'test_prob': 0.16157026250297254}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,416] Trial 35 finished with value: 76.95443646149553 and parameters: {'beta': 0.017644383199111966, 'test_prob': 0.22374415307055945}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,584] Trial 36 finished with value: 101.94377949993742 and parameters: {'beta': 0.01554178388608823, 'test_prob': 0.22409934352533223}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,700] Trial 37 finished with value: 102.34552493559275 and parameters: {'beta': 0.018867781944167765, 'test_prob': 0.2242769794998071}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,869] Trial 38 finished with value: 76.77758209664452 and parameters: {'beta': 0.018970450253246653, 'test_prob': 0.22513134122178088}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:39,983] Trial 39 finished with value: 106.87838362267505 and parameters: {'beta': 0.017522249768622437, 'test_prob': 0.19821481400589852}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:40,156] Trial 40 finished with value: 97.13225772810519 and parameters: {'beta': 0.019914748783776734, 'test_prob': 0.18535151241021924}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:40,268] Trial 41 finished with value: 109.0436970481553 and parameters: {'beta': 0.01983893214009034, 'test_prob': 0.18141179858826403}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:40,436] Trial 42 finished with value: 119.28355901502299 and parameters: {'beta': 0.01863790351975733, 'test_prob': 0.21895390644157595}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:40,552] Trial 43 finished with value: 92.11031420362293 and parameters: {'beta': 0.01868476134268849, 'test_prob': 0.2317985832396006}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:40,716] Trial 44 finished with value: 100.63347831059056 and parameters: {'beta': 0.017814869708936885, 'test_prob': 0.29811837829716237}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:40,833] Trial 45 finished with value: 94.98303182227866 and parameters: {'beta': 0.017610737098589367, 'test_prob': 0.2981166902639556}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,006] Trial 46 finished with value: 100.189972320135 and parameters: {'beta': 0.017209979721453533, 'test_prob': 0.21189355599889337}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,107] Trial 47 finished with value: 89.12443025583713 and parameters: {'beta': 0.016107109834634738, 'test_prob': 0.24559866604046102}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,297] Trial 48 finished with value: 93.38360645711705 and parameters: {'beta': 0.016289517499479527, 'test_prob': 0.02388147288839343}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,390] Trial 49 finished with value: 118.0301933285079 and parameters: {'beta': 0.01889845971925482, 'test_prob': 0.26108979212875255}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,581] Trial 50 finished with value: 78.79117470002286 and parameters: {'beta': 0.01902359300267874, 'test_prob': 0.26099216077708365}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,677] Trial 51 finished with value: 112.04958724576242 and parameters: {'beta': 0.015429386890541536, 'test_prob': 0.14527235557747156}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,867] Trial 52 finished with value: 86.92716624816751 and parameters: {'beta': 0.01930530322477765, 'test_prob': 0.27492060728139167}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:41,970] Trial 53 finished with value: 86.64954216424252 and parameters: {'beta': 0.01931929300664503, 'test_prob': 0.27463577634401565}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:42,154] Trial 54 finished with value: 75.5878779010821 and parameters: {'beta': 0.018210629742400588, 'test_prob': 0.2333708292259026}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:42,256] Trial 55 finished with value: 80.1053348095009 and parameters: {'beta': 0.018156746823482905, 'test_prob': 0.236625588602763}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:42,443] Trial 56 finished with value: 88.56959735132611 and parameters: {'beta': 0.01820573134015419, 'test_prob': 0.2383189444089499}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:42,539] Trial 57 finished with value: 118.03691404662128 and parameters: {'beta': 0.018356290491938477, 'test_prob': 0.19689067796599494}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:42,729] Trial 58 finished with value: 55.25801415322299 and parameters: {'beta': 0.019987256979818412, 'test_prob': 0.1950163316099901}. Best is trial 25 with value: 49.57462390802959.
[I 2022-10-28 22:19:42,796] Trial 59 finished with value: 100.51023618713891 and parameters: {'beta': 0.017167199374469113, 'test_prob': 0.2123249983659167}. Best is trial 25 with value: 49.57462390802959.
Making results structure...
Processed 60 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 60 total trials completed in 9.2 s.

Initial parameter values:
#0. 'beta':      0.013
#1. 'test_prob': 0.01

Best parameter values:
#0. 'beta':      0.019701393540332103
#1. 'test_prob': 0.1748859516818704

Mismatch before calibration: 141.953
Mismatch after calibration:  49.5746
Percent improvement:         65.1%
../_images/tutorials_tut_calibration_18_3.svg
../_images/tutorials_tut_calibration_18_4.svg