T7 - Calibration#

We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.

Click here to open an interactive version of this notebook.

The Fit object#

The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:

[1]:

import covasim as cv
cv.options(jupyter=True, verbose=0)

pars = dict(
    pop_size  = 20_000,
    start_day = '2020-02-01',
    end_day   = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

Covasim 3.1.6 (2024-01-28) — © 2020-2024 by IDM

../_images/tutorials_tut_calibration_3_1.png

We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?

[2]:

fit = sim.compute_fit()
fit.summarize()

Mismatch values for:
#0. 'cum_deaths':    np.float64(55.65217391304348)
#1. 'cum_tests':     np.float64(0.0)
#2. 'cum_diagnoses': np.float64(22.428498467824312)

Total mismatch value:
78.08067238086778

So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)

What if we improve the fit? Does the mismatch reduce?

[3]:

sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim

# Rerun and compute fit
sim.run()
fit = sim.compute_fit()

# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

../_images/tutorials_tut_calibration_7_0.png

[4]:

fit.plot()
fit.summarize()

../_images/tutorials_tut_calibration_8_0.png

Mismatch values for:
#0. 'cum_deaths':    np.float64(43.913043478260875)
#1. 'cum_tests':     np.float64(0.0)
#2. 'cum_diagnoses': np.float64(23.240551583248212)

Total mismatch value:
67.1535950615091

As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:

\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)

where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).

Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.

Calibration approaches#

Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).

In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.

However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:

import covasim as cv
import numpy as np
from scipy import optimize

def objective(x, n_runs=10):
    print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
    pars = dict(
        pop_size       = 20_000,
        start_day      = '2020-02-01',
        end_day        = '2020-04-11',
        beta           = x[0],
        rel_death_prob = x[1],
        verbose        = 0,
    )
    sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
    msim = cv.MultiSim(sim)
    msim.run(n_runs=n_runs)
    mismatches = []
    for sim in msim.sims:
        fit = sim.compute_fit()
        mismatches.append(fit.mismatch)
    mismatch = np.mean(mismatches)
    return mismatch

guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization

This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.

What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.

Built-in calibration#

One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!

[5]:

'''
Example for running built-in calibration with Optuna
'''

import sciris as sc
import covasim as cv

# Create default simulation
pars = sc.objdict(
    pop_size       = 20_000,
    start_day      = '2020-02-01',
    end_day        = '2020-04-11',
    beta           = 0.015,
    rel_death_prob = 1.0,
    interventions  = cv.test_num(daily_tests='data'),
    verbose        = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')

# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
    beta           = [pars.beta, 0.005, 0.20],
    rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)

if __name__ == '__main__':

    # Run the calibration
    calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)

/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-covasim/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
[I 2024-07-02 10:25:24,558] A new study created in RDB with name: covasim_calibration

Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db

[I 2024-07-02 10:25:25,813] Trial 1 finished with value: 653.7183017275836 and parameters: {'beta': 0.0229363859151455, 'rel_death_prob': 1.880577272177021}. Best is trial 1 with value: 653.7183017275836.
[I 2024-07-02 10:25:26,022] Trial 0 finished with value: 873.5768530443665 and parameters: {'beta': 0.19924956494942284, 'rel_death_prob': 0.6160884462694783}. Best is trial 1 with value: 653.7183017275836.
[I 2024-07-02 10:25:27,017] Trial 2 finished with value: 2295.9577208331484 and parameters: {'beta': 0.18022473887822446, 'rel_death_prob': 2.16041236892693}. Best is trial 1 with value: 653.7183017275836.
[I 2024-07-02 10:25:27,186] Trial 3 finished with value: 1197.631012124173 and parameters: {'beta': 0.13037691417148384, 'rel_death_prob': 0.898513212762305}. Best is trial 1 with value: 653.7183017275836.
[I 2024-07-02 10:25:27,758] Trial 4 finished with value: 146.65896877914463 and parameters: {'beta': 0.005401372693901181, 'rel_death_prob': 0.5125305082561156}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:28,228] Trial 5 finished with value: 711.7336901008127 and parameters: {'beta': 0.03704361945257706, 'rel_death_prob': 1.0605438639977574}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:28,899] Trial 6 finished with value: 2222.6356752675756 and parameters: {'beta': 0.12774673445244872, 'rel_death_prob': 2.40326898365786}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:29,416] Trial 7 finished with value: 2031.4473508904382 and parameters: {'beta': 0.1658747054921802, 'rel_death_prob': 1.8294058297083178}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:30,102] Trial 8 finished with value: 2310.5664608962115 and parameters: {'beta': 0.1968202225399829, 'rel_death_prob': 1.6363095496933797}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:30,630] Trial 9 finished with value: 2284.9784607185684 and parameters: {'beta': 0.19569839158183422, 'rel_death_prob': 1.8506702742955157}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:31,299] Trial 10 finished with value: 795.0948172491894 and parameters: {'beta': 0.1908456922915627, 'rel_death_prob': 0.5268794030988195}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:31,722] Trial 11 finished with value: 1187.3306834835903 and parameters: {'beta': 0.06348518807346748, 'rel_death_prob': 1.250237967448797}. Best is trial 4 with value: 146.65896877914463.
[I 2024-07-02 10:25:32,092] Trial 12 finished with value: 132.49944486388063 and parameters: {'beta': 0.009611835352860462, 'rel_death_prob': 1.3390179209456892}. Best is trial 12 with value: 132.49944486388063.
[I 2024-07-02 10:25:32,520] Trial 13 finished with value: 118.15816938313273 and parameters: {'beta': 0.01053136518698631, 'rel_death_prob': 2.989227864575076}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:32,891] Trial 14 finished with value: 127.58016165563795 and parameters: {'beta': 0.009718930965135476, 'rel_death_prob': 1.4098464985983277}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:33,614] Trial 15 finished with value: 2016.2584935826264 and parameters: {'beta': 0.06641579811987258, 'rel_death_prob': 2.9338177894332387}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:33,986] Trial 16 finished with value: 1729.0511613447616 and parameters: {'beta': 0.059910170085517606, 'rel_death_prob': 2.9208229979251388}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:34,705] Trial 17 finished with value: 1925.5545809832572 and parameters: {'beta': 0.057298786673049364, 'rel_death_prob': 2.9151177016296375}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:35,043] Trial 18 finished with value: 1443.9551227961097 and parameters: {'beta': 0.03890321988072477, 'rel_death_prob': 2.499851658486908}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:35,759] Trial 19 finished with value: 1359.7498556646092 and parameters: {'beta': 0.0316699851603031, 'rel_death_prob': 2.4566952349085716}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:36,180] Trial 20 finished with value: 1692.1640316205533 and parameters: {'beta': 0.09501270520733471, 'rel_death_prob': 1.484713110440548}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:36,870] Trial 21 finished with value: 1507.468801350091 and parameters: {'beta': 0.08687151924796471, 'rel_death_prob': 1.548045648617016}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:36,965] Trial 22 finished with value: 137.92201447795 and parameters: {'beta': 0.006356012280451793, 'rel_death_prob': 1.3724346860867174}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:37,675] Trial 23 finished with value: 129.40467202558068 and parameters: {'beta': 0.007500544773441246, 'rel_death_prob': 1.4109089398966483}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:37,889] Trial 24 finished with value: 140.48818670337965 and parameters: {'beta': 0.017507819397546337, 'rel_death_prob': 1.0394519230681554}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:38,751] Trial 25 finished with value: 827.75469645157 and parameters: {'beta': 0.046712818556873814, 'rel_death_prob': 0.992953918147162}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:38,963] Trial 26 finished with value: 1334.6295021539281 and parameters: {'beta': 0.04435017270495615, 'rel_death_prob': 2.1468697705987054}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:39,775] Trial 27 finished with value: 673.8509792601146 and parameters: {'beta': 0.024628376645305962, 'rel_death_prob': 2.23091652375785}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:40,061] Trial 28 finished with value: 1791.9445307989522 and parameters: {'beta': 0.07973247551251958, 'rel_death_prob': 2.0820391169894408}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:40,879] Trial 29 finished with value: 2227.0194741750674 and parameters: {'beta': 0.07889959095790586, 'rel_death_prob': 2.652513046466238}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:41,211] Trial 30 finished with value: 2336.292578940356 and parameters: {'beta': 0.11960687857573057, 'rel_death_prob': 2.6698604954971317}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:42,028] Trial 31 finished with value: 1067.458919927166 and parameters: {'beta': 0.11734215335793668, 'rel_death_prob': 0.863050801693712}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:42,163] Trial 32 finished with value: 192.23231336323667 and parameters: {'beta': 0.018074083304424123, 'rel_death_prob': 0.7767239308625135}. Best is trial 13 with value: 118.15816938313273.
[I 2024-07-02 10:25:42,885] Trial 33 finished with value: 60.81771550384155 and parameters: {'beta': 0.01566546690585519, 'rel_death_prob': 1.3242022867379022}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:42,997] Trial 34 finished with value: 108.16249944486387 and parameters: {'beta': 0.015073925216000447, 'rel_death_prob': 1.241080795534311}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:43,899] Trial 35 finished with value: 645.4020295776525 and parameters: {'beta': 0.026876680239109708, 'rel_death_prob': 1.1581775503503}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:44,089] Trial 36 finished with value: 742.5632855176088 and parameters: {'beta': 0.02864001928185412, 'rel_death_prob': 1.2330167016495623}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:45,083] Trial 37 finished with value: 1940.2752364879868 and parameters: {'beta': 0.1558227084616245, 'rel_death_prob': 1.6283907016001533}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:45,179] Trial 38 finished with value: 1137.4444863880624 and parameters: {'beta': 0.04935930679894904, 'rel_death_prob': 1.6770539326288054}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:46,090] Trial 40 finished with value: 90.08859972465248 and parameters: {'beta': 0.01631860679898218, 'rel_death_prob': 2.027492785196057}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:46,169] Trial 39 finished with value: 588.0880001776435 and parameters: {'beta': 0.047938062287675984, 'rel_death_prob': 0.7099371634849325}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:47,035] Trial 41 finished with value: 160.0697250965937 and parameters: {'beta': 0.017259608580302288, 'rel_death_prob': 2.0002263254342703}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:47,138] Trial 42 finished with value: 262.3836434693787 and parameters: {'beta': 0.01871377646182012, 'rel_death_prob': 1.9427546404883054}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:47,946] Trial 43 finished with value: 103.09921392725497 and parameters: {'beta': 0.016125325195036167, 'rel_death_prob': 1.7467306895570194}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:48,206] Trial 44 finished with value: 1105.1512190789183 and parameters: {'beta': 0.03611673427751026, 'rel_death_prob': 1.7792440458464356}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:49,016] Trial 45 finished with value: 1062.6765332859618 and parameters: {'beta': 0.03618034575400436, 'rel_death_prob': 1.786668626430834}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:49,252] Trial 46 finished with value: 1309.2576719811698 and parameters: {'beta': 0.03256332701517326, 'rel_death_prob': 2.7557427733598354}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:50,055] Trial 47 finished with value: 447.9116889461296 and parameters: {'beta': 0.021044807034044204, 'rel_death_prob': 2.2802168055811505}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:50,131] Trial 48 finished with value: 86.00157658657902 and parameters: {'beta': 0.015222247502193878, 'rel_death_prob': 1.150816939124606}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:50,999] Trial 50 finished with value: 67.70440111915443 and parameters: {'beta': 0.013731582137301294, 'rel_death_prob': 1.1755426128785003}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:51,217] Trial 49 finished with value: 1519.2706621663633 and parameters: {'beta': 0.14816465531396616, 'rel_death_prob': 1.1783724371930613}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:52,074] Trial 52 finished with value: 75.02797886041657 and parameters: {'beta': 0.01321404984511125, 'rel_death_prob': 0.9290156288394531}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:52,084] Trial 51 finished with value: 910.7562064218146 and parameters: {'beta': 0.055818436744626596, 'rel_death_prob': 1.0904470609504289}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:52,862] Trial 53 finished with value: 146.65386152684638 and parameters: {'beta': 0.005507948497560164, 'rel_death_prob': 0.9367422195037118}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:52,863] Trial 54 finished with value: 146.65896877914463 and parameters: {'beta': 0.005384411245844078, 'rel_death_prob': 0.8982244432427691}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:53,926] Trial 55 finished with value: 439.2548963005729 and parameters: {'beta': 0.024974508974260432, 'rel_death_prob': 0.7260320152138544}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:53,930] Trial 56 finished with value: 338.16227739041614 and parameters: {'beta': 0.023105712797865136, 'rel_death_prob': 0.5984358043044744}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:55,028] Trial 57 finished with value: 1072.6256828174269 and parameters: {'beta': 0.041645290088217525, 'rel_death_prob': 1.4953881273774434}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:55,031] Trial 58 finished with value: 1066.8739174845673 and parameters: {'beta': 0.040955105209354276, 'rel_death_prob': 1.520134175460333}. Best is trial 33 with value: 60.81771550384155.
[I 2024-07-02 10:25:55,897] Trial 59 finished with value: 75.54725318648133 and parameters: {'beta': 0.013964270463063127, 'rel_death_prob': 1.336096703536969}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:55,899] Trial 60 finished with value: 47.1701381178665 and parameters: {'beta': 0.01316966931262417, 'rel_death_prob': 1.3464584255297767}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:56,743] Trial 61 finished with value: 119.02140604876317 and parameters: {'beta': 0.010859698821933154, 'rel_death_prob': 1.0811004596472524}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:56,745] Trial 62 finished with value: 126.52695740995692 and parameters: {'beta': 0.011990270962570836, 'rel_death_prob': 1.2964523225660731}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:57,830] Trial 63 finished with value: 886.6849491495313 and parameters: {'beta': 0.0307219359834112, 'rel_death_prob': 1.3415821716874896}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:57,830] Trial 64 finished with value: 752.7479237909135 and parameters: {'beta': 0.02879109116146509, 'rel_death_prob': 1.3527049728756297}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:58,673] Trial 66 finished with value: 130.76431140915753 and parameters: {'beta': 0.011863165708511751, 'rel_death_prob': 1.1387875880992149}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:58,684] Trial 65 finished with value: 94.70588888395434 and parameters: {'beta': 0.012449175629299725, 'rel_death_prob': 1.1508498016566984}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:59,734] Trial 68 finished with value: 547.947772793889 and parameters: {'beta': 0.02283831331403374, 'rel_death_prob': 1.4426655169698333}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:25:59,752] Trial 67 finished with value: 955.5392592263623 and parameters: {'beta': 0.035144034768507276, 'rel_death_prob': 1.4450629074356027}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:00,966] Trial 69 finished with value: 1402.4430430341522 and parameters: {'beta': 0.17664836455815586, 'rel_death_prob': 0.9867571545623293}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:01,002] Trial 70 finished with value: 1308.9153750499622 and parameters: {'beta': 0.18013974024892587, 'rel_death_prob': 1.0040374728030899}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:01,841] Trial 71 finished with value: 89.24012967979749 and parameters: {'beta': 0.01291535040850493, 'rel_death_prob': 1.273519273005343}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:01,847] Trial 72 finished with value: 87.62390638184483 and parameters: {'beta': 0.014151378350278927, 'rel_death_prob': 1.5917224476581935}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:02,899] Trial 73 finished with value: 593.7153039925389 and parameters: {'beta': 0.022851771716661223, 'rel_death_prob': 1.5807858075059658}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:02,908] Trial 74 finished with value: 466.3041257716392 and parameters: {'beta': 0.02276212588967986, 'rel_death_prob': 1.292196282701037}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:03,724] Trial 75 finished with value: 138.1714926499978 and parameters: {'beta': 0.005749302574377373, 'rel_death_prob': 1.2309970861112676}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:03,730] Trial 76 finished with value: 143.6176000355287 and parameters: {'beta': 0.006251389279397125, 'rel_death_prob': 0.8243988019266484}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:04,603] Trial 78 finished with value: 69.30508060576453 and parameters: {'beta': 0.013466097920750857, 'rel_death_prob': 1.6749403107118193}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:04,831] Trial 77 finished with value: 1105.904538792912 and parameters: {'beta': 0.05232080914605765, 'rel_death_prob': 1.3940361245661514}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:05,667] Trial 79 finished with value: 879.7250965936848 and parameters: {'beta': 0.029800730467857195, 'rel_death_prob': 1.713715704270872}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:05,888] Trial 80 finished with value: 833.4063152284941 and parameters: {'beta': 0.028772593000151097, 'rel_death_prob': 1.6570938829662105}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:06,616] Trial 81 finished with value: 167.7767908691211 and parameters: {'beta': 0.01790896072989308, 'rel_death_prob': 1.5813973321934407}. Best is trial 60 with value: 47.1701381178665.
[I 2024-07-02 10:26:06,745] Trial 82 finished with value: 39.95880889994227 and parameters: {'beta': 0.013806787549379381, 'rel_death_prob': 1.568172228804058}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:07,454] Trial 83 finished with value: 61.33454723098103 and parameters: {'beta': 0.013092700402248638, 'rel_death_prob': 1.2949986006202003}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:07,584] Trial 84 finished with value: 70.44044499711329 and parameters: {'beta': 0.014647825785880685, 'rel_death_prob': 1.6004769721248355}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:08,571] Trial 85 finished with value: 1099.2587822534085 and parameters: {'beta': 0.07000085085731826, 'rel_death_prob': 1.1794820045825745}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:08,576] Trial 86 finished with value: 257.8591730692366 and parameters: {'beta': 0.020108707200269423, 'rel_death_prob': 1.4817222788225233}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:09,410] Trial 87 finished with value: 121.64209264111558 and parameters: {'beta': 0.011020278732760347, 'rel_death_prob': 1.9097462111670669}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:09,715] Trial 88 finished with value: 1793.0827818981213 and parameters: {'beta': 0.10556300937163991, 'rel_death_prob': 1.8941305340080818}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:10,468] Trial 89 finished with value: 1070.9347381978062 and parameters: {'beta': 0.03374162560695846, 'rel_death_prob': 1.8439834243172366}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:10,775] Trial 90 finished with value: 806.2541635208954 and parameters: {'beta': 0.033608196204692777, 'rel_death_prob': 1.422050514181416}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:11,503] Trial 91 finished with value: 742.9566549717991 and parameters: {'beta': 0.025490859778034974, 'rel_death_prob': 1.4198273966605321}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:11,703] Trial 92 finished with value: 135.79151307900696 and parameters: {'beta': 0.01727560582351058, 'rel_death_prob': 1.3272831140474273}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:12,451] Trial 93 finished with value: 149.75218723631036 and parameters: {'beta': 0.016844480113883408, 'rel_death_prob': 1.216386646659941}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:12,519] Trial 94 finished with value: 130.90254030288227 and parameters: {'beta': 0.008898148326970107, 'rel_death_prob': 1.5346355700147325}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:13,277] Trial 95 finished with value: 125.3222010036861 and parameters: {'beta': 0.010139080515750663, 'rel_death_prob': 1.0905926788705576}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:13,355] Trial 96 finished with value: 134.02329351156905 and parameters: {'beta': 0.00942401955027166, 'rel_death_prob': 1.0409836678807303}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:14,284] Trial 97 finished with value: 347.2157703068793 and parameters: {'beta': 0.020377653976775288, 'rel_death_prob': 1.693973718235806}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:14,375] Trial 98 finished with value: 331.94919394235467 and parameters: {'beta': 0.020350748639387788, 'rel_death_prob': 1.7172887370450958}. Best is trial 82 with value: 39.95880889994227.
[I 2024-07-02 10:26:14,965] Trial 99 finished with value: 940.0811609006529 and parameters: {'beta': 0.039092250682038365, 'rel_death_prob': 1.3538234719264468}. Best is trial 82 with value: 39.95880889994227.

Making results structure...
Processed 100 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 100 total trials completed in 51.4 s.

Initial parameter values:
#0. 'beta':           0.015
#1. 'rel_death_prob': 1.0

Best parameter values:
#0. 'beta':           0.013806787549379381
#1. 'rel_death_prob': 1.568172228804058

Mismatch before calibration: 101.465
Mismatch after calibration:  39.9588
Percent improvement:         60.6%

So it improved the fit (see above), but let’s visualize this as a plot:

[6]:

# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

../_images/tutorials_tut_calibration_16_0.png

../_images/tutorials_tut_calibration_16_1.png

Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.

Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.

[7]:

pars = dict(
    verbose = 0,
    start_day = '2020-02-05',
    pop_size = 1000,
    pop_scale = 4,
    interventions = cv.test_prob(symp_prob=0.01),
)

sim = cv.Sim(pars, datafile='example_data.csv')

calib_pars = dict(
    beta      = [0.013, 0.005, 0.020],
    test_prob = [0.01, 0.00, 0.30]
)

def set_test_prob(sim, calib_pars):
    tp = sim.get_intervention(cv.test_prob)
    tp.symp_prob = calib_pars['test_prob']
    return sim

calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])

[I 2024-07-02 10:26:17,642] A new study created in RDB with name: covasim_calibration

Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db

[I 2024-07-02 10:26:18,087] Trial 0 finished with value: 104.81160891346488 and parameters: {'beta': 0.01668972794572236, 'test_prob': 0.026043377903012366}. Best is trial 0 with value: 104.81160891346488.
[I 2024-07-02 10:26:18,116] Trial 1 finished with value: 58.6779317470115 and parameters: {'beta': 0.018556162078596387, 'test_prob': 0.06862160309632202}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:18,358] Trial 2 finished with value: 144.799086516009 and parameters: {'beta': 0.006440325056727441, 'test_prob': 0.14370006379288194}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:18,424] Trial 3 finished with value: 109.29513660207243 and parameters: {'beta': 0.013468157311738546, 'test_prob': 0.011712818791519308}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:18,620] Trial 4 finished with value: 134.7897171736733 and parameters: {'beta': 0.010016097865356607, 'test_prob': 0.2193638764249073}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:18,699] Trial 5 finished with value: 113.58625487952867 and parameters: {'beta': 0.012027445180290976, 'test_prob': 0.09658802368893578}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:18,882] Trial 6 finished with value: 126.28388658111507 and parameters: {'beta': 0.010788175706814259, 'test_prob': 0.25884836071115663}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:18,967] Trial 7 finished with value: 131.63093006441548 and parameters: {'beta': 0.013087554493945688, 'test_prob': 0.1060173268589503}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:19,156] Trial 8 finished with value: 69.49706075292626 and parameters: {'beta': 0.016739386426737897, 'test_prob': 0.038435475259085645}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:19,239] Trial 9 finished with value: 107.4243642205878 and parameters: {'beta': 0.01813093404329502, 'test_prob': 0.18813372951879767}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:19,426] Trial 10 finished with value: 107.4974098136436 and parameters: {'beta': 0.018007068403646073, 'test_prob': 0.1909015671427018}. Best is trial 1 with value: 58.6779317470115.
[I 2024-07-02 10:26:19,536] Trial 11 finished with value: 49.17040743604218 and parameters: {'beta': 0.018892255012471222, 'test_prob': 0.08159170362818428}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:19,714] Trial 12 finished with value: 89.44992977745221 and parameters: {'beta': 0.019473209214854365, 'test_prob': 0.06360637017774555}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:19,825] Trial 13 finished with value: 130.55098696465728 and parameters: {'beta': 0.019511875903576635, 'test_prob': 0.07436183529804877}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:19,995] Trial 14 finished with value: 128.71947201786813 and parameters: {'beta': 0.015482150588750554, 'test_prob': 0.0908237009812182}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,115] Trial 15 finished with value: 132.60804566034307 and parameters: {'beta': 0.015113802590007759, 'test_prob': 0.14121555379906775}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,283] Trial 16 finished with value: 101.99087791977846 and parameters: {'beta': 0.015114738100122739, 'test_prob': 0.1374535180910102}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,394] Trial 17 finished with value: 140.71116281110827 and parameters: {'beta': 0.008018253748987472, 'test_prob': 0.123484348147302}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,526] Trial 18 finished with value: 145.74961694199567 and parameters: {'beta': 0.0072577237481680585, 'test_prob': 0.059103014593259326}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,688] Trial 19 finished with value: 102.43706037571127 and parameters: {'beta': 0.019762502508211, 'test_prob': 0.04621270108425153}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,810] Trial 20 finished with value: 118.8908422693761 and parameters: {'beta': 0.01970608651310086, 'test_prob': 0.2942282783667}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:20,981] Trial 21 finished with value: 110.65699987043227 and parameters: {'beta': 0.017804471139490113, 'test_prob': 0.17599180536462303}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,101] Trial 22 finished with value: 125.32672332729462 and parameters: {'beta': 0.017502056366687786, 'test_prob': 0.0007730253024989253}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,269] Trial 23 finished with value: 134.89399818150952 and parameters: {'beta': 0.016583513779680707, 'test_prob': 0.002258369209011929}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,393] Trial 24 finished with value: 132.77371667260175 and parameters: {'beta': 0.016289151377313935, 'test_prob': 0.04150069741502058}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,559] Trial 25 finished with value: 64.65789689805536 and parameters: {'beta': 0.016401318868941934, 'test_prob': 0.0395604623635352}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,679] Trial 26 finished with value: 112.06811461547014 and parameters: {'beta': 0.014415925206274703, 'test_prob': 0.08014882328926529}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,843] Trial 27 finished with value: 85.57368896275948 and parameters: {'beta': 0.013866253775629559, 'test_prob': 0.08085751660782751}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:21,973] Trial 28 finished with value: 111.95148735448387 and parameters: {'beta': 0.019026987963911623, 'test_prob': 0.11076441493996585}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:22,131] Trial 29 finished with value: 70.96290128436729 and parameters: {'beta': 0.018390506737876535, 'test_prob': 0.11381250561118228}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:22,267] Trial 30 finished with value: 118.81413826228149 and parameters: {'beta': 0.01706291647107173, 'test_prob': 0.03162947313406568}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:22,423] Trial 31 finished with value: 99.33929110504445 and parameters: {'beta': 0.016932909152800435, 'test_prob': 0.023802796610673888}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:22,566] Trial 32 finished with value: 101.47308328409335 and parameters: {'beta': 0.016146424742736438, 'test_prob': 0.024603720930128824}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:22,708] Trial 33 finished with value: 119.53184677040299 and parameters: {'beta': 0.016080014198804914, 'test_prob': 0.05358803220394778}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:22,833] Trial 34 finished with value: 145.23070130983814 and parameters: {'beta': 0.005130830808611471, 'test_prob': 0.05642587910746408}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,003] Trial 35 finished with value: 82.11497021165667 and parameters: {'beta': 0.0186591695269631, 'test_prob': 0.03592037281631257}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,121] Trial 36 finished with value: 83.75244005137196 and parameters: {'beta': 0.018668464269410517, 'test_prob': 0.03367527091670516}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,291] Trial 37 finished with value: 125.1334776545731 and parameters: {'beta': 0.011579214344408503, 'test_prob': 0.02294496704974694}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,404] Trial 38 finished with value: 126.86890029315707 and parameters: {'beta': 0.011880398173536356, 'test_prob': 0.16166224245962332}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,573] Trial 39 finished with value: 133.80388770842328 and parameters: {'beta': 0.012818464351356147, 'test_prob': 0.07117669055711422}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,679] Trial 40 finished with value: 142.24961012569034 and parameters: {'beta': 0.009682487439035058, 'test_prob': 0.08827904225143818}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,866] Trial 41 finished with value: 115.47637915869461 and parameters: {'beta': 0.017451738216307536, 'test_prob': 0.09367159907293207}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:23,975] Trial 42 finished with value: 65.55011531622009 and parameters: {'beta': 0.018360476604960103, 'test_prob': 0.11317692802369315}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:24,157] Trial 43 finished with value: 85.25529242130445 and parameters: {'beta': 0.018438179903503174, 'test_prob': 0.11274436371512012}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:24,267] Trial 44 finished with value: 110.16241450369075 and parameters: {'beta': 0.018955468596308304, 'test_prob': 0.10310049229702961}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:24,449] Trial 45 finished with value: 98.1209087485453 and parameters: {'beta': 0.019065890393066533, 'test_prob': 0.06556737963801222}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:24,556] Trial 46 finished with value: 53.9542981162615 and parameters: {'beta': 0.015662407531450692, 'test_prob': 0.12465986790499592}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:24,740] Trial 47 finished with value: 77.20391147474533 and parameters: {'beta': 0.01763747415972432, 'test_prob': 0.21145394579776197}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:24,849] Trial 48 finished with value: 90.0784054793383 and parameters: {'beta': 0.014396289751498342, 'test_prob': 0.12872314380231484}. Best is trial 11 with value: 49.17040743604218.
[I 2024-07-02 10:26:25,027] Trial 49 finished with value: 47.33750493896927 and parameters: {'beta': 0.01559136944667151, 'test_prob': 0.12677056140642856}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:25,139] Trial 50 finished with value: 81.91845649225102 and parameters: {'beta': 0.015788159697715083, 'test_prob': 0.13146774355164872}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:25,319] Trial 51 finished with value: 117.45878115223712 and parameters: {'beta': 0.01558158592310129, 'test_prob': 0.15945142275145707}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:25,428] Trial 52 finished with value: 121.69186031046641 and parameters: {'beta': 0.015287033371462137, 'test_prob': 0.15116726162718963}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:25,606] Trial 53 finished with value: 97.72816311442735 and parameters: {'beta': 0.01481415941968825, 'test_prob': 0.12305051309493713}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:25,732] Trial 54 finished with value: 116.23451518709126 and parameters: {'beta': 0.014676229491847551, 'test_prob': 0.11778598930604986}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:25,907] Trial 55 finished with value: 83.06224737731695 and parameters: {'beta': 0.019954999926019554, 'test_prob': 0.10440855922810072}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:26,029] Trial 56 finished with value: 67.1346785313942 and parameters: {'beta': 0.01713028158969467, 'test_prob': 0.09650964414626678}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:26,192] Trial 57 finished with value: 131.06803345564677 and parameters: {'beta': 0.013741433560811877, 'test_prob': 0.08448049018327228}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:26,323] Trial 58 finished with value: 83.5872112743141 and parameters: {'beta': 0.01794462337058607, 'test_prob': 0.14838529758127567}. Best is trial 49 with value: 47.33750493896927.
[I 2024-07-02 10:26:26,430] Trial 59 finished with value: 75.20990425768986 and parameters: {'beta': 0.01809295758103144, 'test_prob': 0.14376989241684168}. Best is trial 49 with value: 47.33750493896927.

Making results structure...
Processed 60 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 60 total trials completed in 9.2 s.

Initial parameter values:
#0. 'beta':      0.013
#1. 'test_prob': 0.01

Best parameter values:
#0. 'beta':      0.01559136944667151
#1. 'test_prob': 0.12677056140642856

Mismatch before calibration: 141.953
Mismatch after calibration:  47.3375
Percent improvement:         66.7%

../_images/tutorials_tut_calibration_18_4.png

../_images/tutorials_tut_calibration_18_5.png