T7 - Calibration

We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.

Click here to open an interactive version of this notebook.

The Fit object

The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:

[1]:
import covasim as cv
cv.options(jupyter=True, verbose=0)

pars = dict(
    pop_size  = 20_000,
    start_day = '2020-02-01',
    end_day   = '2020-04-11',
    beta      = 0.015,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
Covasim 3.1.2 (2022-01-16) — © 2020-2022 by IDM
../_images/tutorials_tut_calibration_3_1.svg

We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?

[2]:
fit = sim.compute_fit()
print(fit.mismatches)
print(fit.mismatch)
#0. 'cum_deaths':    84.78260869565216
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 16.682839632277833
101.46544832792999

So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)

What if we improve the fit? Does the mismatch reduce?

[3]:
sim['rel_death_prob'] = 2 # Double the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim

# Rerun and compute fit
sim.run()
fit = sim.compute_fit()

# Output
sim.plot()
fit.plot()
print(fit.mismatches)
print(fit.mismatch)
../_images/tutorials_tut_calibration_7_0.svg
../_images/tutorials_tut_calibration_7_1.svg
#0. 'cum_deaths':    78.2608695652174
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 16.544943820224717
94.80581338544212

As expected, the fit is improved.

Calibration approaches

Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).

In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.

However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:

import covasim as cv
import numpy as np
import scipy

def objective(x, n_runs=10):
    print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
    pars = dict(
        pop_size       = 20_000,
        start_day      = '2020-02-01',
        end_day        = '2020-04-11',
        beta           = x[0],
        rel_death_prob = x[1],
        verbose        = 0,
    )
    sim = cv.Sim(pars=pars, datafile='/home/cliffk/idm/covasim/docs/tutorials/example_data.csv', interventions=cv.test_num(daily_tests='data'))
    msim = cv.MultiSim(sim)
    msim.run(n_runs=n_runs)
    mismatches = []
    for sim in msim.sims:
        fit = sim.compute_fit()
        mismatches.append(fit.mismatch)
    mismatch = np.mean(mismatches)
    return mismatch

guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = scipy.optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization

This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.

What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.

Built-in calibration

One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!

[4]:
'''
Example for running built-in calibration with Optuna
'''

import sciris as sc
import covasim as cv

# Create default simulation
pars = sc.objdict(
    pop_size       = 20_000,
    start_day      = '2020-02-01',
    end_day        = '2020-04-11',
    beta           = 0.015,
    rel_death_prob = 1.0,
    interventions  = cv.test_num(daily_tests='data'),
    verbose        = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')

# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
    beta           = [pars.beta, 0.005, 0.20],
    rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)

if __name__ == '__main__':

    # Run the calibration
    n_trials = 20
    n_workers = 4
    calib = sim.calibrate(calib_pars=calib_pars, n_trials=n_trials, n_workers=n_workers)
/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-covasim/envs/latest/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
[I 2022-05-02 22:49:33,908] A new study created in RDB with name: covasim_calibration
[I 2022-05-02 22:49:35,627] Trial 0 finished with value: 1421.7527867833192 and parameters: {'beta': 0.1415262728990144, 'rel_death_prob': 1.2015492243330765}. Best is trial 0 with value: 1421.7527867833192.
[I 2022-05-02 22:49:35,637] Trial 1 finished with value: 2269.925278678332 and parameters: {'beta': 0.14061960979125446, 'rel_death_prob': 2.3958424363349935}. Best is trial 0 with value: 1421.7527867833192.
[I 2022-05-02 22:49:36,621] Trial 3 finished with value: 142.39952036239285 and parameters: {'beta': 0.009203114995062042, 'rel_death_prob': 1.1392772416772337}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:36,956] Trial 2 finished with value: 700.7895145889772 and parameters: {'beta': 0.028047971041828202, 'rel_death_prob': 1.4671125982597233}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:38,192] Trial 4 finished with value: 943.1014344717324 and parameters: {'beta': 0.18507235747853984, 'rel_death_prob': 0.5821169583933543}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:38,405] Trial 5 finished with value: 1895.7758582404406 and parameters: {'beta': 0.09232309473543646, 'rel_death_prob': 2.355893997479659}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:39,687] Trial 6 finished with value: 2179.7548518896833 and parameters: {'beta': 0.11070179966709182, 'rel_death_prob': 2.082475609615379}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:39,892] Trial 7 finished with value: 1906.8806901452235 and parameters: {'beta': 0.11007635124841912, 'rel_death_prob': 2.0553393410820817}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:41,264] Trial 8 finished with value: 2568.973775369721 and parameters: {'beta': 0.18550238717172782, 'rel_death_prob': 2.0869598275645176}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:41,436] Trial 9 finished with value: 2609.057045787627 and parameters: {'beta': 0.14600481028853896, 'rel_death_prob': 2.5685637718576606}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:42,224] Trial 10 finished with value: 143.47137718168494 and parameters: {'beta': 0.006562094431753005, 'rel_death_prob': 1.021813450543688}. Best is trial 3 with value: 142.39952036239285.
[I 2022-05-02 22:49:42,368] Trial 11 finished with value: 141.3400985921748 and parameters: {'beta': 0.005260247462726006, 'rel_death_prob': 0.7891975709180737}. Best is trial 11 with value: 141.3400985921748.
[I 2022-05-02 22:49:43,192] Trial 12 finished with value: 139.00763867300262 and parameters: {'beta': 0.007204959840590469, 'rel_death_prob': 0.8875996915310891}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:43,753] Trial 13 finished with value: 550.3572856064307 and parameters: {'beta': 0.05006360067483297, 'rel_death_prob': 0.5002334854008443}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:44,610] Trial 14 finished with value: 561.9487498334591 and parameters: {'beta': 0.056590700165255824, 'rel_death_prob': 0.5025096795840087}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:45,172] Trial 15 finished with value: 808.4427321579251 and parameters: {'beta': 0.05752810539132515, 'rel_death_prob': 0.8796867693071144}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:46,018] Trial 16 finished with value: 955.262690411689 and parameters: {'beta': 0.05695077502392047, 'rel_death_prob': 0.9291059430926423}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:46,551] Trial 17 finished with value: 1179.8839765510502 and parameters: {'beta': 0.03395153423872806, 'rel_death_prob': 2.974103266663434}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:47,369] Trial 18 finished with value: 858.9723320158104 and parameters: {'beta': 0.030634612532272544, 'rel_death_prob': 1.5666961436186388}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:48,010] Trial 19 finished with value: 1648.2485455433673 and parameters: {'beta': 0.08754942202503796, 'rel_death_prob': 1.7080128611514367}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:48,801] Trial 20 finished with value: 1505.227938890616 and parameters: {'beta': 0.0828071272775918, 'rel_death_prob': 1.3662243005230055}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:48,929] Trial 21 finished with value: 141.3400985921748 and parameters: {'beta': 0.005298380345316324, 'rel_death_prob': 1.3251205077456172}. Best is trial 12 with value: 139.00763867300262.
[I 2022-05-02 22:49:49,790] Trial 22 finished with value: 120.25991473109207 and parameters: {'beta': 0.010931864439821413, 'rel_death_prob': 0.7949937775807762}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:49,866] Trial 23 finished with value: 144.1627437047564 and parameters: {'beta': 0.00603625927046752, 'rel_death_prob': 0.7669338465584574}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:51,043] Trial 24 finished with value: 311.48387884709325 and parameters: {'beta': 0.020439280883884678, 'rel_death_prob': 1.2385994848994608}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:51,182] Trial 25 finished with value: 473.1660523160279 and parameters: {'beta': 0.025106385316825717, 'rel_death_prob': 0.7682059985891072}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:52,420] Trial 26 finished with value: 658.4006528400764 and parameters: {'beta': 0.039328925497002516, 'rel_death_prob': 0.6957477138794603}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:52,568] Trial 27 finished with value: 805.7366656304125 and parameters: {'beta': 0.04282337368507103, 'rel_death_prob': 1.032653589710466}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:53,885] Trial 28 finished with value: 1069.8917484567216 and parameters: {'beta': 0.07623403098107115, 'rel_death_prob': 1.0069902852065127}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:54,021] Trial 29 finished with value: 1350.6943642581161 and parameters: {'beta': 0.06856991135606577, 'rel_death_prob': 1.3326466278063671}. Best is trial 22 with value: 120.25991473109207.
[I 2022-05-02 22:49:55,069] Trial 31 finished with value: 110.60654172403073 and parameters: {'beta': 0.014504538126386527, 'rel_death_prob': 0.6987217822903403}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:55,307] Trial 30 finished with value: 1121.0271128480704 and parameters: {'beta': 0.07101722076859504, 'rel_death_prob': 1.1732122033463532}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:56,149] Trial 32 finished with value: 147.13361016121155 and parameters: {'beta': 0.01703755015652529, 'rel_death_prob': 1.1693149189003207}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:56,597] Trial 33 finished with value: 182.49400452991074 and parameters: {'beta': 0.01833948894086213, 'rel_death_prob': 0.6647043719736004}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:57,346] Trial 34 finished with value: 180.74976684282987 and parameters: {'beta': 0.01916029510733827, 'rel_death_prob': 0.6907937251140213}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:57,798] Trial 35 finished with value: 174.44775058844428 and parameters: {'beta': 0.0188248438212765, 'rel_death_prob': 1.562466197936955}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:58,734] Trial 36 finished with value: 703.4113114535685 and parameters: {'beta': 0.039515633634655226, 'rel_death_prob': 0.8451536493843228}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:49:59,317] Trial 37 finished with value: 1156.1708931029889 and parameters: {'beta': 0.13446084174504658, 'rel_death_prob': 0.8995698961644284}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:50:00,268] Trial 38 finished with value: 1442.8879291202204 and parameters: {'beta': 0.15205070631283887, 'rel_death_prob': 1.0198022257552575}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:50:00,688] Trial 39 finished with value: 980.332082426611 and parameters: {'beta': 0.030844454984230343, 'rel_death_prob': 1.8384708024609933}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:50:01,709] Trial 40 finished with value: 476.55971044100016 and parameters: {'beta': 0.029169399699637923, 'rel_death_prob': 0.6108768376124014}. Best is trial 31 with value: 110.60654172403073.
[I 2022-05-02 22:50:01,804] Trial 41 finished with value: 108.69420882000266 and parameters: {'beta': 0.012858350340728332, 'rel_death_prob': 0.6069557528212473}. Best is trial 41 with value: 108.69420882000266.
[I 2022-05-02 22:50:02,700] Trial 42 finished with value: 120.76830838921704 and parameters: {'beta': 0.010085114794223032, 'rel_death_prob': 0.7926082487236688}. Best is trial 41 with value: 108.69420882000266.
[I 2022-05-02 22:50:02,823] Trial 43 finished with value: 106.27348225784962 and parameters: {'beta': 0.012878725051824745, 'rel_death_prob': 0.8018220456547198}. Best is trial 43 with value: 106.27348225784962.
[I 2022-05-02 22:50:03,711] Trial 44 finished with value: 101.51596571479327 and parameters: {'beta': 0.015436348930858792, 'rel_death_prob': 0.6158110133477049}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:03,845] Trial 45 finished with value: 110.18963449837901 and parameters: {'beta': 0.014522072727294482, 'rel_death_prob': 0.6015743326868733}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:05,276] Trial 46 finished with value: 920.4769729537684 and parameters: {'beta': 0.1959553769226354, 'rel_death_prob': 0.5749996484703619}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:05,429] Trial 47 finished with value: 814.4594084469512 and parameters: {'beta': 0.19642663051861659, 'rel_death_prob': 0.5688241406875384}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:06,657] Trial 48 finished with value: 466.1064973131412 and parameters: {'beta': 0.045938269513442424, 'rel_death_prob': 0.5725000160630604}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:06,837] Trial 49 finished with value: 572.7327130612426 and parameters: {'beta': 0.04469675207888847, 'rel_death_prob': 0.5086895684343073}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:08,140] Trial 50 finished with value: 909.0599324954479 and parameters: {'beta': 0.10652869045102284, 'rel_death_prob': 0.6989431807677471}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:08,326] Trial 51 finished with value: 801.7431274148421 and parameters: {'beta': 0.09715198800276008, 'rel_death_prob': 0.6657283130118935}. Best is trial 44 with value: 101.51596571479327.
[I 2022-05-02 22:50:09,379] Trial 53 finished with value: 58.125194297641784 and parameters: {'beta': 0.01379169842570969, 'rel_death_prob': 1.0986712119216342}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:09,443] Trial 52 finished with value: 554.9941155571346 and parameters: {'beta': 0.0253430565106538, 'rel_death_prob': 1.060983658190446}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:10,457] Trial 55 finished with value: 99.20782075764977 and parameters: {'beta': 0.01410250625173605, 'rel_death_prob': 0.922225921796844}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:10,655] Trial 54 finished with value: 385.4915175200959 and parameters: {'beta': 0.022837684320989763, 'rel_death_prob': 0.9552145155624285}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:11,850] Trial 56 finished with value: 664.325176533286 and parameters: {'beta': 0.034266658623902296, 'rel_death_prob': 0.9535735233525634}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:12,044] Trial 57 finished with value: 644.2479015854688 and parameters: {'beta': 0.032492128534556484, 'rel_death_prob': 0.8547586045187027}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:13,272] Trial 58 finished with value: 638.0347071101835 and parameters: {'beta': 0.050825872687963805, 'rel_death_prob': 0.8294346444086951}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:13,455] Trial 59 finished with value: 1591.2860283341474 and parameters: {'beta': 0.052001489693141535, 'rel_death_prob': 2.177740565615122}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:14,283] Trial 60 finished with value: 100.80550250921527 and parameters: {'beta': 0.012945785190752537, 'rel_death_prob': 1.120101106497079}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:14,962] Trial 61 finished with value: 1260.4277878935914 and parameters: {'beta': 0.12047283669105364, 'rel_death_prob': 1.1274476232298216}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:15,301] Trial 62 finished with value: 63.89550117688857 and parameters: {'beta': 0.013276610998202565, 'rel_death_prob': 1.4407902911094335}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:15,988] Trial 63 finished with value: 89.53701647644003 and parameters: {'beta': 0.01298885248090163, 'rel_death_prob': 1.268290420157232}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:16,305] Trial 64 finished with value: 86.68883510236711 and parameters: {'beta': 0.012831379988131662, 'rel_death_prob': 1.2526830666433715}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:17,318] Trial 65 finished with value: 545.62141937203 and parameters: {'beta': 0.026107684652395748, 'rel_death_prob': 1.2610737963715664}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:17,596] Trial 66 finished with value: 419.1943864635609 and parameters: {'beta': 0.02294163931596488, 'rel_death_prob': 1.2460803201846156}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:18,549] Trial 68 finished with value: 144.1627437047564 and parameters: {'beta': 0.005930891396822692, 'rel_death_prob': 1.4676222306740667}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:18,672] Trial 67 finished with value: 860.6290802504774 and parameters: {'beta': 0.03588201849651454, 'rel_death_prob': 1.439462494102156}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:19,908] Trial 69 finished with value: 821.94619620731 and parameters: {'beta': 0.03569840066562641, 'rel_death_prob': 1.4417422575053844}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:20,215] Trial 70 finished with value: 1555.8832215659281 and parameters: {'beta': 0.16578984630981175, 'rel_death_prob': 1.1026777063986946}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:21,237] Trial 72 finished with value: 110.799507039126 and parameters: {'beta': 0.01242146059949847, 'rel_death_prob': 1.302530272975372}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:21,312] Trial 71 finished with value: 1473.0397033352579 and parameters: {'beta': 0.0634378522847518, 'rel_death_prob': 1.7245026457900228}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:22,245] Trial 73 finished with value: 110.047186570147 and parameters: {'beta': 0.01067234208316473, 'rel_death_prob': 1.5925367732781115}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:22,335] Trial 74 finished with value: 68.65423901940756 and parameters: {'beta': 0.011528631646593748, 'rel_death_prob': 1.6086431814638227}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:23,466] Trial 75 finished with value: 214.40822489674468 and parameters: {'beta': 0.01871293492950881, 'rel_death_prob': 1.1871053679373453}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:23,570] Trial 76 finished with value: 352.3009281875916 and parameters: {'beta': 0.020304873653735428, 'rel_death_prob': 1.9189318434178853}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:24,384] Trial 77 finished with value: 141.3400985921748 and parameters: {'beta': 0.005144149755918382, 'rel_death_prob': 1.8784490268729555}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:24,886] Trial 78 finished with value: 808.4025403028822 and parameters: {'beta': 0.027121658392214427, 'rel_death_prob': 1.6206872852845116}. Best is trial 53 with value: 58.125194297641784.
[I 2022-05-02 22:50:25,430] Trial 79 finished with value: 660.8580183861083 and parameters: {'beta': 0.02742377248240772, 'rel_death_prob': 1.5117266950343695}. Best is trial 53 with value: 58.125194297641784.
Making results structure...
Processed 80 trials; 0 failed
Removed existing calibration covasim_calibration.db
Calibration for 80 total trials completed in 51.7 s.

Initial parameter values:
#0. 'beta':           0.015
#1. 'rel_death_prob': 1.0

Best parameter values:
#0. 'beta':           0.01379169842570969
#1. 'rel_death_prob': 1.0986712119216342

Mismatch before calibration: 101.465
Mismatch after calibration:  58.1252
Percent improvement:         42.7%

So it improved the fit (see above), but let’s visualize this as a plot:

[5]:
# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
../_images/tutorials_tut_calibration_15_0.svg
../_images/tutorials_tut_calibration_15_1.svg
../_images/tutorials_tut_calibration_15_2.svg

Compared to scipy.optimize.minimize(), Optuna took less time and produced a much better fit. However, it’s still far from perfect – more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.

Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.

[6]:
pars = dict(
    verbose = 0,
    start_day = '2020-02-05',
    pop_size = 1000,
    pop_scale = 4,
    interventions = cv.test_prob(symp_prob=0.01),
)

sim = cv.Sim(pars, datafile='example_data.csv')

calib_pars = dict(
    beta      = [0.013, 0.005, 0.020],
    test_prob = [0.01, 0.00, 0.30]
)

def set_test_prob(sim, calib_pars):
    tp = sim.get_intervention(cv.test_prob)
    tp.symp_prob = calib_pars['test_prob']
    return sim

calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, n_trials=5)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])
[I 2022-05-02 22:50:29,281] A new study created in RDB with name: covasim_calibration
[I 2022-05-02 22:50:29,747] Trial 0 finished with value: 102.08701354375941 and parameters: {'beta': 0.01799459931237144, 'test_prob': 0.1588753800764354}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:29,756] Trial 1 finished with value: 135.7039909918339 and parameters: {'beta': 0.012945198514329185, 'test_prob': 0.08983663418049712}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,064] Trial 2 finished with value: 106.58415601934192 and parameters: {'beta': 0.015362032118758952, 'test_prob': 0.2888666400312742}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,064] Trial 3 finished with value: 144.41712375457567 and parameters: {'beta': 0.005757190736086227, 'test_prob': 0.2628239409268523}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,367] Trial 4 finished with value: 139.70010330451424 and parameters: {'beta': 0.009465504764148997, 'test_prob': 0.20045223467548387}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,422] Trial 5 finished with value: 120.5511599530964 and parameters: {'beta': 0.01616821306041767, 'test_prob': 0.18922955286174317}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,669] Trial 6 finished with value: 143.53078787306796 and parameters: {'beta': 0.012425531260558696, 'test_prob': 0.24575364547632125}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,724] Trial 7 finished with value: 143.86328146530997 and parameters: {'beta': 0.007503190149167631, 'test_prob': 0.22928813119505098}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:30,995] Trial 8 finished with value: 105.39999546492669 and parameters: {'beta': 0.011213191540166414, 'test_prob': 0.2912984082357921}. Best is trial 0 with value: 102.08701354375941.
[I 2022-05-02 22:50:31,039] Trial 9 finished with value: 84.40449665902221 and parameters: {'beta': 0.019318278842670407, 'test_prob': 0.09791502180726216}. Best is trial 9 with value: 84.40449665902221.
Making results structure...
Processed 10 trials; 0 failed
Removed existing calibration covasim_calibration.db
Calibration for 10 total trials completed in 1.9 s.

Initial parameter values:
#0. 'beta':      0.013
#1. 'test_prob': 0.01

Best parameter values:
#0. 'beta':      0.019318278842670407
#1. 'test_prob': 0.09791502180726216

Mismatch before calibration: 141.953
Mismatch after calibration:  84.4045
Percent improvement:         40.5%
../_images/tutorials_tut_calibration_17_2.svg
../_images/tutorials_tut_calibration_17_3.svg
../_images/tutorials_tut_calibration_17_4.svg