T7 - Calibration

We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.

Click here to open an interactive version of this notebook.

The Fit object

The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:

import covasim as cv
cv.options(jupyter=True, verbose=0)

pars = dict(
    pop_size  = 20_000,
    start_day = '2020-02-01',
    end_day   = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

Covasim 3.1.6 (2024-01-28) — © 2020-2024 by IDM

We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?

fit = sim.compute_fit()
fit.summarize()

Mismatch values for:
#0. 'cum_deaths':    55.65217391304348
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 22.428498467824312

Total mismatch value:
78.08067238086778

So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)

What if we improve the fit? Does the mismatch reduce?

sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim

# Rerun and compute fit
sim.run()
fit = sim.compute_fit()

# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

fit.plot()
fit.summarize()

Mismatch values for:
#0. 'cum_deaths':    43.913043478260875
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 23.240551583248212

Total mismatch value:
67.1535950615091

As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:

\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)

where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).

Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.

Calibration approaches

Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).

In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.

However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:

import covasim as cv
import numpy as np
from scipy import optimize

def objective(x, n_runs=10):
    print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
    pars = dict(
        pop_size       = 20_000,
        start_day      = '2020-02-01',
        end_day        = '2020-04-11',
        beta           = x[0],
        rel_death_prob = x[1],
        verbose        = 0,
    )
    sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
    msim = cv.MultiSim(sim)
    msim.run(n_runs=n_runs)
    mismatches = []
    for sim in msim.sims:
        fit = sim.compute_fit()
        mismatches.append(fit.mismatch)
    mismatch = np.mean(mismatches)
    return mismatch

guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization

This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.

What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.

Built-in calibration

One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!

'''
Example for running built-in calibration with Optuna
'''

import sciris as sc
import covasim as cv

# Create default simulation
pars = sc.objdict(
    pop_size       = 20_000,
    start_day      = '2020-02-01',
    end_day        = '2020-04-11',
    beta           = 0.015,
    rel_death_prob = 1.0,
    interventions  = cv.test_num(daily_tests='data'),
    verbose        = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')

# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
    beta           = [pars.beta, 0.005, 0.20],
    rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)

if __name__ == '__main__':

    # Run the calibration
    calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)

/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-covasim/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db

[I 2024-03-15 03:42:42,521] A new study created in RDB with name: covasim_calibration
[I 2024-03-15 03:42:44,079] Trial 0 finished with value: 1666.7301372296486 and parameters: {'beta': 0.12259791765742985, 'rel_death_prob': 1.5721248815265894}. Best is trial 0 with value: 1666.7301372296486.
[I 2024-03-15 03:42:44,087] Trial 1 finished with value: 1975.0166540835814 and parameters: {'beta': 0.0835681682446089, 'rel_death_prob': 2.348977521936377}. Best is trial 0 with value: 1666.7301372296486.
[I 2024-03-15 03:42:45,423] Trial 2 finished with value: 1022.3392325798286 and parameters: {'beta': 0.12666028555554787, 'rel_death_prob': 0.9208062743704138}. Best is trial 2 with value: 1022.3392325798286.
[I 2024-03-15 03:42:45,533] Trial 3 finished with value: 2842.4274992228097 and parameters: {'beta': 0.1807759734518775, 'rel_death_prob': 2.6327378310194347}. Best is trial 2 with value: 1022.3392325798286.
[I 2024-03-15 03:42:46,362] Trial 4 finished with value: 138.01394501931873 and parameters: {'beta': 0.006301471964836163, 'rel_death_prob': 1.4903703013540102}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:47,007] Trial 5 finished with value: 2754.068481591686 and parameters: {'beta': 0.1951140834280025, 'rel_death_prob': 2.515617299027657}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:47,744] Trial 6 finished with value: 1256.613336590132 and parameters: {'beta': 0.11596756494430475, 'rel_death_prob': 1.3533590474818922}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:48,504] Trial 7 finished with value: 2950.9131989163743 and parameters: {'beta': 0.1953473119045991, 'rel_death_prob': 2.972160466298388}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:49,204] Trial 8 finished with value: 2154.7129946262826 and parameters: {'beta': 0.14659162394276115, 'rel_death_prob': 1.897679171066921}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:49,901] Trial 9 finished with value: 2212.038792912022 and parameters: {'beta': 0.1254684351988183, 'rel_death_prob': 2.704776356641281}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:50,558] Trial 10 finished with value: 2033.9231469556337 and parameters: {'beta': 0.06220275651250387, 'rel_death_prob': 2.9255659457042835}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:51,151] Trial 11 finished with value: 262.6712039792157 and parameters: {'beta': 0.021071801915582986, 'rel_death_prob': 0.5622777298845438}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:51,556] Trial 12 finished with value: 141.01545498956344 and parameters: {'beta': 0.007229594809471329, 'rel_death_prob': 0.6618709202931039}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:52,079] Trial 13 finished with value: 142.29260114580094 and parameters: {'beta': 0.00500550924473801, 'rel_death_prob': 0.5032119619352105}. Best is trial 4 with value: 138.01394501931873.
[I 2024-03-15 03:42:52,585] Trial 14 finished with value: 97.80776746458233 and parameters: {'beta': 0.012769480110763592, 'rel_death_prob': 1.1117725751258916}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:53,398] Trial 15 finished with value: 932.3083670115911 and parameters: {'beta': 0.04582463626290263, 'rel_death_prob': 1.1848421692832805}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:53,939] Trial 16 finished with value: 886.9732868499357 and parameters: {'beta': 0.042552767220613866, 'rel_death_prob': 1.1737366532158724}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:54,701] Trial 17 finished with value: 1287.4005196074077 and parameters: {'beta': 0.04329845430785164, 'rel_death_prob': 1.960521941680108}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:55,324] Trial 18 finished with value: 1751.5993471599236 and parameters: {'beta': 0.0781748293522335, 'rel_death_prob': 1.9489497939976017}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:56,098] Trial 19 finished with value: 1611.3464271439357 and parameters: {'beta': 0.09337242828970545, 'rel_death_prob': 1.5637904677865546}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:56,610] Trial 20 finished with value: 330.9391792867611 and parameters: {'beta': 0.021744553801901236, 'rel_death_prob': 1.627423629043936}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:57,379] Trial 21 finished with value: 641.7495669938269 and parameters: {'beta': 0.02931694565441305, 'rel_death_prob': 0.9098983269125783}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:57,615] Trial 22 finished with value: 119.78782697517431 and parameters: {'beta': 0.01238843678332088, 'rel_death_prob': 0.8225235390029892}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:58,359] Trial 23 finished with value: 138.1714926499978 and parameters: {'beta': 0.005708615307318915, 'rel_death_prob': 0.7883702265145807}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:59,000] Trial 24 finished with value: 786.5533818892393 and parameters: {'beta': 0.06357308474112205, 'rel_death_prob': 0.8453131683343874}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:42:59,688] Trial 25 finished with value: 1104.4746191766221 and parameters: {'beta': 0.06524051364953176, 'rel_death_prob': 1.151246300109355}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:43:00,326] Trial 26 finished with value: 909.4562996846827 and parameters: {'beta': 0.031180090765675024, 'rel_death_prob': 1.172268843640524}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:43:00,994] Trial 27 finished with value: 863.3946573699872 and parameters: {'beta': 0.03183166294536136, 'rel_death_prob': 1.3770520737357517}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:43:01,691] Trial 28 finished with value: 1113.359905848914 and parameters: {'beta': 0.05123695461803793, 'rel_death_prob': 1.3964288811955299}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:43:02,192] Trial 29 finished with value: 148.74061819958254 and parameters: {'beta': 0.01776623293770952, 'rel_death_prob': 2.171376232023894}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:43:02,860] Trial 30 finished with value: 123.6455788959453 and parameters: {'beta': 0.01667389392305108, 'rel_death_prob': 1.0497718350355052}. Best is trial 14 with value: 97.80776746458233.
[I 2024-03-15 03:43:03,278] Trial 31 finished with value: 84.97535195629969 and parameters: {'beta': 0.01537429744791513, 'rel_death_prob': 1.0331535203121516}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:03,966] Trial 32 finished with value: 98.75360838477594 and parameters: {'beta': 0.015115699937224412, 'rel_death_prob': 1.0245725531508152}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:04,538] Trial 33 finished with value: 275.29422214326956 and parameters: {'beta': 0.02060882506134022, 'rel_death_prob': 1.0870086509645434}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:05,281] Trial 34 finished with value: 673.3389217036017 and parameters: {'beta': 0.03478787792695419, 'rel_death_prob': 0.7159037700130727}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:05,852] Trial 35 finished with value: 580.7340009770396 and parameters: {'beta': 0.03571773084204952, 'rel_death_prob': 0.7174229688739544}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:06,748] Trial 36 finished with value: 1264.9870098148067 and parameters: {'beta': 0.1475849722568023, 'rel_death_prob': 0.9475172001917749}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:07,216] Trial 37 finished with value: 965.2548074787939 and parameters: {'beta': 0.05355109777586674, 'rel_death_prob': 0.9930224358099857}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:08,083] Trial 38 finished with value: 1005.3034374028512 and parameters: {'beta': 0.05064273407290009, 'rel_death_prob': 1.2725556767491417}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:08,613] Trial 39 finished with value: 1353.0713460940622 and parameters: {'beta': 0.07350446387838447, 'rel_death_prob': 1.2542061469886938}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:09,499] Trial 40 finished with value: 1512.574610294444 and parameters: {'beta': 0.08144697294444965, 'rel_death_prob': 1.6701963225343026}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:10,011] Trial 41 finished with value: 1588.4861438024602 and parameters: {'beta': 0.10667086755270294, 'rel_death_prob': 1.7673374596322642}. Best is trial 31 with value: 84.97535195629969.
[I 2024-03-15 03:43:10,575] Trial 42 finished with value: 69.76817515654838 and parameters: {'beta': 0.015640716728713852, 'rel_death_prob': 1.0519018025067126}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:11,087] Trial 43 finished with value: 82.30459208597948 and parameters: {'beta': 0.013953297683639163, 'rel_death_prob': 1.000075473608534}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:11,677] Trial 44 finished with value: 80.90353954789714 and parameters: {'beta': 0.0159534509486743, 'rel_death_prob': 0.8427490622609369}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:12,376] Trial 45 finished with value: 722.2559621619221 and parameters: {'beta': 0.024992710575437706, 'rel_death_prob': 1.471980876907254}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:12,963] Trial 46 finished with value: 666.6536394723987 and parameters: {'beta': 0.026177226910857478, 'rel_death_prob': 1.4754235658552655}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:13,845] Trial 47 finished with value: 868.9936492427943 and parameters: {'beta': 0.15949047465692112, 'rel_death_prob': 0.5842956361332198}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:14,471] Trial 48 finished with value: 852.3461162677088 and parameters: {'beta': 0.16732408234634677, 'rel_death_prob': 0.5972000473163777}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:15,176] Trial 49 finished with value: 708.3846649198384 and parameters: {'beta': 0.03814796404849967, 'rel_death_prob': 0.8854221177206176}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:15,800] Trial 50 finished with value: 664.1202202780122 and parameters: {'beta': 0.039906484022764616, 'rel_death_prob': 0.9433444780607758}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:16,243] Trial 51 finished with value: 93.16416485322202 and parameters: {'beta': 0.012678639069257008, 'rel_death_prob': 1.2915325240056805}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:16,851] Trial 52 finished with value: 116.32866278811565 and parameters: {'beta': 0.010735080501005533, 'rel_death_prob': 1.0908639242579454}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:17,289] Trial 53 finished with value: 124.576209086468 and parameters: {'beta': 0.011120540625393471, 'rel_death_prob': 1.273126161456969}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:17,917] Trial 54 finished with value: 80.40502731269709 and parameters: {'beta': 0.013101537153884782, 'rel_death_prob': 0.7350405045868674}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:18,613] Trial 55 finished with value: 497.101079184616 and parameters: {'beta': 0.02730978961587168, 'rel_death_prob': 0.7480115745698079}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:19,236] Trial 56 finished with value: 458.6160456543945 and parameters: {'beta': 0.02437000854780717, 'rel_death_prob': 0.7378288085505793}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:19,873] Trial 57 finished with value: 257.5495181418484 and parameters: {'beta': 0.02030263249712172, 'rel_death_prob': 0.8299836690858325}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:20,182] Trial 58 finished with value: 145.42301372296487 and parameters: {'beta': 0.005897848388910434, 'rel_death_prob': 0.6189660838428079}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:20,866] Trial 59 finished with value: 132.7207221210641 and parameters: {'beta': 0.00743740749483608, 'rel_death_prob': 0.5099933806907004}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:21,271] Trial 60 finished with value: 101.10483190478305 and parameters: {'beta': 0.015423754013697881, 'rel_death_prob': 1.0025492211985245}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:21,933] Trial 61 finished with value: 105.84613847315362 and parameters: {'beta': 0.014819766460652879, 'rel_death_prob': 0.9644633820116412}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:22,355] Trial 62 finished with value: 80.06073189145978 and parameters: {'beta': 0.01583987626833108, 'rel_death_prob': 0.8897525698482875}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:23,233] Trial 63 finished with value: 752.3303504019186 and parameters: {'beta': 0.03166942733481917, 'rel_death_prob': 1.1418204412686155}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:23,666] Trial 64 finished with value: 769.9445974152862 and parameters: {'beta': 0.04616681864780176, 'rel_death_prob': 0.8650915266231761}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:24,591] Trial 65 finished with value: 901.3167828751609 and parameters: {'beta': 0.04724171918944962, 'rel_death_prob': 0.8775171308885827}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:24,936] Trial 66 finished with value: 295.552027357108 and parameters: {'beta': 0.021721123201949938, 'rel_death_prob': 0.6515746031754455}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:25,762] Trial 67 finished with value: 306.53317493449396 and parameters: {'beta': 0.021696038007144175, 'rel_death_prob': 0.6649069996212538}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:25,800] Trial 68 finished with value: 141.64908735621975 and parameters: {'beta': 0.005059545968978903, 'rel_death_prob': 1.2380973631796246}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:26,727] Trial 69 finished with value: 88.48725407469911 and parameters: {'beta': 0.012784940956941661, 'rel_death_prob': 1.3503274541036947}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:27,047] Trial 70 finished with value: 1108.8930585779633 and parameters: {'beta': 0.058431539507105085, 'rel_death_prob': 1.3461345768886404}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:28,106] Trial 72 finished with value: 129.78505129457744 and parameters: {'beta': 0.011760122517449378, 'rel_death_prob': 1.0459179949849664}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:28,152] Trial 71 finished with value: 949.5114802149487 and parameters: {'beta': 0.11750713172614403, 'rel_death_prob': 0.7848171560320244}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:29,304] Trial 73 finished with value: 111.57658657902917 and parameters: {'beta': 0.016761810630282432, 'rel_death_prob': 1.2080411886058187}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:29,597] Trial 74 finished with value: 1564.5663276635432 and parameters: {'beta': 0.13152007740199786, 'rel_death_prob': 1.352209877907862}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:30,643] Trial 75 finished with value: 823.4862548296842 and parameters: {'beta': 0.02975924654323192, 'rel_death_prob': 1.3413210299656195}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:30,952] Trial 76 finished with value: 728.156170893103 and parameters: {'beta': 0.032299207995731896, 'rel_death_prob': 1.0847127202108227}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:31,964] Trial 77 finished with value: 834.3383887729271 and parameters: {'beta': 0.03468055872207575, 'rel_death_prob': 1.0716835615219413}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:32,234] Trial 78 finished with value: 489.9218368343918 and parameters: {'beta': 0.026180298851424942, 'rel_death_prob': 0.9361002051922104}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:33,289] Trial 80 finished with value: 121.35897322023361 and parameters: {'beta': 0.011167354585250736, 'rel_death_prob': 1.5843372610562232}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:33,352] Trial 79 finished with value: 1050.03752720167 and parameters: {'beta': 0.09091530885737178, 'rel_death_prob': 0.9339573807319759}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:34,507] Trial 82 finished with value: 103.2751920770973 and parameters: {'beta': 0.016180525409888378, 'rel_death_prob': 1.136273454258094}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:34,601] Trial 81 finished with value: 1481.2228538437626 and parameters: {'beta': 0.040920912216973686, 'rel_death_prob': 2.35453842340012}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:35,532] Trial 83 finished with value: 133.48492250299773 and parameters: {'beta': 0.01042979344320219, 'rel_death_prob': 2.1089745421655235}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:35,653] Trial 84 finished with value: 126.4011635653062 and parameters: {'beta': 0.012036633268333359, 'rel_death_prob': 0.8016115044100607}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:36,735] Trial 85 finished with value: 145.7132388861749 and parameters: {'beta': 0.01733143439848604, 'rel_death_prob': 0.827272725313267}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:36,872] Trial 86 finished with value: 228.0307101301239 and parameters: {'beta': 0.01954718297165229, 'rel_death_prob': 1.4097176857756306}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:38,020] Trial 87 finished with value: 685.2656881467336 and parameters: {'beta': 0.023039033883652568, 'rel_death_prob': 2.839819961637692}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:38,166] Trial 88 finished with value: 465.7342230314874 and parameters: {'beta': 0.023861173476483578, 'rel_death_prob': 1.0059396012457065}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:39,535] Trial 89 finished with value: 1692.7229426655417 and parameters: {'beta': 0.19986171214736842, 'rel_death_prob': 1.1500337761173904}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:39,688] Trial 90 finished with value: 1770.5543589288095 and parameters: {'beta': 0.18641358968768618, 'rel_death_prob': 1.3077989468345668}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:40,534] Trial 91 finished with value: 135.6283030599103 and parameters: {'beta': 0.00848005797639188, 'rel_death_prob': 1.2778519374983988}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:40,691] Trial 92 finished with value: 140.6834835901763 and parameters: {'beta': 0.008581477737011007, 'rel_death_prob': 1.0147571389683634}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:41,477] Trial 93 finished with value: 141.64908735621975 and parameters: {'beta': 0.005058112750138991, 'rel_death_prob': 1.0326440715652976}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:41,731] Trial 94 finished with value: 111.11460230048408 and parameters: {'beta': 0.0150803530197699, 'rel_death_prob': 1.1195432175729452}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:42,520] Trial 95 finished with value: 84.55711240396144 and parameters: {'beta': 0.014603206654681476, 'rel_death_prob': 1.201501430601762}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:43,044] Trial 96 finished with value: 603.0059510591998 and parameters: {'beta': 0.02826568603969007, 'rel_death_prob': 1.1931635734637578}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:43,836] Trial 97 finished with value: 700.3950348625483 and parameters: {'beta': 0.027937767060143295, 'rel_death_prob': 1.4419414593697493}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:44,222] Trial 98 finished with value: 949.2577830083937 and parameters: {'beta': 0.037817266264098454, 'rel_death_prob': 1.4402825069710996}. Best is trial 42 with value: 69.76817515654838.
[I 2024-03-15 03:43:44,911] Trial 99 finished with value: 169.89141537504997 and parameters: {'beta': 0.01894566079783324, 'rel_death_prob': 0.9051430114658252}. Best is trial 42 with value: 69.76817515654838.

Making results structure...
Processed 100 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 100 total trials completed in 63.3 s.

Initial parameter values:
#0. 'beta':           0.015
#1. 'rel_death_prob': 1.0

Best parameter values:
#0. 'beta':           0.015640716728713852
#1. 'rel_death_prob': 1.0519018025067126

Mismatch before calibration: 101.465
Mismatch after calibration:  69.7682
Percent improvement:         31.2%

So it improved the fit (see above), but let’s visualize this as a plot:

# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.

Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.

pars = dict(
    verbose = 0,
    start_day = '2020-02-05',
    pop_size = 1000,
    pop_scale = 4,
    interventions = cv.test_prob(symp_prob=0.01),
)

sim = cv.Sim(pars, datafile='example_data.csv')

calib_pars = dict(
    beta      = [0.013, 0.005, 0.020],
    test_prob = [0.01, 0.00, 0.30]
)

def set_test_prob(sim, calib_pars):
    tp = sim.get_intervention(cv.test_prob)
    tp.symp_prob = calib_pars['test_prob']
    return sim

calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])

Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db

[I 2024-03-15 03:43:48,657] A new study created in RDB with name: covasim_calibration
[I 2024-03-15 03:43:49,235] Trial 0 finished with value: 128.1000462677516 and parameters: {'beta': 0.010906521109987945, 'test_prob': 0.00023412842410638345}. Best is trial 0 with value: 128.1000462677516.
[I 2024-03-15 03:43:49,245] Trial 1 finished with value: 113.12288096819128 and parameters: {'beta': 0.012299294981877616, 'test_prob': 0.1195144897631745}. Best is trial 1 with value: 113.12288096819128.
[I 2024-03-15 03:43:49,570] Trial 2 finished with value: 96.40745060012642 and parameters: {'beta': 0.017713995182716477, 'test_prob': 0.26269151333125534}. Best is trial 2 with value: 96.40745060012642.
[I 2024-03-15 03:43:49,596] Trial 3 finished with value: 140.3684457496303 and parameters: {'beta': 0.010056578221081958, 'test_prob': 0.11608650032298924}. Best is trial 2 with value: 96.40745060012642.
[I 2024-03-15 03:43:49,952] Trial 4 finished with value: 95.11804934030883 and parameters: {'beta': 0.014227056512180779, 'test_prob': 0.19143204275041995}. Best is trial 4 with value: 95.11804934030883.
[I 2024-03-15 03:43:50,001] Trial 5 finished with value: 54.90826566663566 and parameters: {'beta': 0.019702862594085116, 'test_prob': 0.2338325840478298}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:50,326] Trial 6 finished with value: 137.98211330621288 and parameters: {'beta': 0.012349893563329736, 'test_prob': 0.11221515242539887}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:50,346] Trial 7 finished with value: 143.8622224254734 and parameters: {'beta': 0.007043781597590117, 'test_prob': 0.23138532229453618}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:50,688] Trial 8 finished with value: 135.818335706053 and parameters: {'beta': 0.016436430627208662, 'test_prob': 0.007957711755556828}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:50,739] Trial 9 finished with value: 78.37649393885712 and parameters: {'beta': 0.013539785796782694, 'test_prob': 0.12326259293975278}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:51,079] Trial 10 finished with value: 95.88329413842449 and parameters: {'beta': 0.018164698038227477, 'test_prob': 0.29925701507142155}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:51,168] Trial 11 finished with value: 79.06855062584478 and parameters: {'beta': 0.019981159042918046, 'test_prob': 0.2771960446405813}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:51,446] Trial 12 finished with value: 109.0436970481553 and parameters: {'beta': 0.019762289697040254, 'test_prob': 0.18138839050161842}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:51,511] Trial 13 finished with value: 145.18729117128612 and parameters: {'beta': 0.00538287114024504, 'test_prob': 0.17813758595753562}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:51,801] Trial 14 finished with value: 144.83397528034462 and parameters: {'beta': 0.0054153626423608225, 'test_prob': 0.07263171554663911}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:51,896] Trial 15 finished with value: 96.07159052349387 and parameters: {'beta': 0.015036580928675224, 'test_prob': 0.06323242688758315}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:52,175] Trial 16 finished with value: 115.38247842155116 and parameters: {'beta': 0.01532242144503335, 'test_prob': 0.2329586704270513}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:52,314] Trial 17 finished with value: 136.50326760230462 and parameters: {'beta': 0.008667778782809235, 'test_prob': 0.22709153392539133}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:52,559] Trial 18 finished with value: 142.8740480581124 and parameters: {'beta': 0.007604321908063303, 'test_prob': 0.21883434496435303}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:52,709] Trial 19 finished with value: 119.90888785575098 and parameters: {'beta': 0.017564515697660335, 'test_prob': 0.14555231683211828}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:52,951] Trial 20 finished with value: 96.58032718809119 and parameters: {'beta': 0.013708688227459869, 'test_prob': 0.1510573843061657}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:53,102] Trial 21 finished with value: 85.8222626558095 and parameters: {'beta': 0.013226096512806698, 'test_prob': 0.08051356929120494}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:53,346] Trial 22 finished with value: 83.9255284290484 and parameters: {'beta': 0.019958105595825826, 'test_prob': 0.24467254041251377}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:53,512] Trial 23 finished with value: 97.75056056720862 and parameters: {'beta': 0.019501055013786857, 'test_prob': 0.29455026466266837}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:53,743] Trial 24 finished with value: 114.82398667216216 and parameters: {'beta': 0.018629509997897, 'test_prob': 0.2829025241751551}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:53,912] Trial 25 finished with value: 97.42744680602208 and parameters: {'beta': 0.01856377288067248, 'test_prob': 0.2694731701896198}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:54,149] Trial 26 finished with value: 112.35471223156362 and parameters: {'beta': 0.016472173840136564, 'test_prob': 0.2646817796563617}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:54,303] Trial 27 finished with value: 132.4811378988817 and parameters: {'beta': 0.016223069974843192, 'test_prob': 0.2044813964443744}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:54,534] Trial 28 finished with value: 134.07310684623977 and parameters: {'beta': 0.016405199500412287, 'test_prob': 0.20529192293958873}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:54,666] Trial 29 finished with value: 137.62103529709833 and parameters: {'beta': 0.010877353106044721, 'test_prob': 0.1441237045630678}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:54,882] Trial 30 finished with value: 104.46485343027263 and parameters: {'beta': 0.011548675569674118, 'test_prob': 0.16448410236528255}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:55,052] Trial 31 finished with value: 125.51016510843743 and parameters: {'beta': 0.017303906926669093, 'test_prob': 0.024991814502087928}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:55,281] Trial 32 finished with value: 79.30032382980453 and parameters: {'beta': 0.019816780186386652, 'test_prob': 0.24225303027798517}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:55,460] Trial 33 finished with value: 116.24299505541913 and parameters: {'beta': 0.01983897297689319, 'test_prob': 0.25501045149070795}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:55,676] Trial 34 finished with value: 82.79878381205042 and parameters: {'beta': 0.01888997472554529, 'test_prob': 0.25640210798380286}. Best is trial 5 with value: 54.90826566663566.
[I 2024-03-15 03:43:55,861] Trial 35 finished with value: 51.35837727157165 and parameters: {'beta': 0.018903570702701943, 'test_prob': 0.09536381005175404}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:56,065] Trial 36 finished with value: 79.14958280015767 and parameters: {'beta': 0.017504019831811466, 'test_prob': 0.27945307078413306}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:56,256] Trial 37 finished with value: 116.87878659136612 and parameters: {'beta': 0.01734954834612025, 'test_prob': 0.08824551625901458}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:56,462] Trial 38 finished with value: 123.8429640593106 and parameters: {'beta': 0.01485539802864419, 'test_prob': 0.10480575847561725}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:56,663] Trial 39 finished with value: 91.50802149732898 and parameters: {'beta': 0.018885981139713494, 'test_prob': 0.11911627541526944}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:56,852] Trial 40 finished with value: 132.2759545729313 and parameters: {'beta': 0.019178732569420164, 'test_prob': 0.05368188702416802}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:57,014] Trial 41 finished with value: 145.02697550462796 and parameters: {'beta': 0.009367457615713063, 'test_prob': 0.05021577388356944}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:57,213] Trial 42 finished with value: 113.65177821837185 and parameters: {'beta': 0.01792652322604872, 'test_prob': 0.27796439454207006}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:57,391] Trial 43 finished with value: 115.37193819416437 and parameters: {'beta': 0.017898281370870427, 'test_prob': 0.2812786563363258}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:57,603] Trial 44 finished with value: 71.70255824766642 and parameters: {'beta': 0.018158023222489727, 'test_prob': 0.10040338407119981}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:57,787] Trial 45 finished with value: 88.90719704118622 and parameters: {'beta': 0.017020230815627754, 'test_prob': 0.09422041228323672}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:57,988] Trial 46 finished with value: 71.71268364993136 and parameters: {'beta': 0.016841664479058733, 'test_prob': 0.10201686756234633}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:58,171] Trial 47 finished with value: 119.96445527153944 and parameters: {'beta': 0.015876577473403125, 'test_prob': 0.12790170789689204}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:58,362] Trial 48 finished with value: 53.210658749220556 and parameters: {'beta': 0.015616843153630575, 'test_prob': 0.12504361907314374}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:58,560] Trial 49 finished with value: 66.05684662093431 and parameters: {'beta': 0.01823867857265475, 'test_prob': 0.13042988369513725}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:58,734] Trial 50 finished with value: 106.64268310983189 and parameters: {'beta': 0.015579824606204575, 'test_prob': 0.1013468538551738}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:58,961] Trial 51 finished with value: 86.40793968282632 and parameters: {'beta': 0.018170407544579003, 'test_prob': 0.1326259747519445}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:59,132] Trial 52 finished with value: 54.645290295222054 and parameters: {'beta': 0.01847876140758688, 'test_prob': 0.1312361268872001}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:59,351] Trial 53 finished with value: 60.88738355314982 and parameters: {'beta': 0.016983451241160165, 'test_prob': 0.11359027677781502}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:59,507] Trial 54 finished with value: 79.49447684149987 and parameters: {'beta': 0.01832646301375519, 'test_prob': 0.155241969438768}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:59,731] Trial 55 finished with value: 120.67840062213779 and parameters: {'beta': 0.01436835079487939, 'test_prob': 0.16162838332726487}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:43:59,871] Trial 56 finished with value: 84.34108457640538 and parameters: {'beta': 0.014625769645505354, 'test_prob': 0.13598225657204152}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:44:00,124] Trial 57 finished with value: 74.14806741213798 and parameters: {'beta': 0.019135800810462626, 'test_prob': 0.11432786845792293}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:44:00,276] Trial 58 finished with value: 67.84993747028483 and parameters: {'beta': 0.01922121381958391, 'test_prob': 0.17689017229313403}. Best is trial 35 with value: 51.35837727157165.
[I 2024-03-15 03:44:00,426] Trial 59 finished with value: 105.42530714767703 and parameters: {'beta': 0.019253401466596338, 'test_prob': 0.17051672072513602}. Best is trial 35 with value: 51.35837727157165.

Making results structure...
Processed 60 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 60 total trials completed in 12.5 s.

Initial parameter values:
#0. 'beta':      0.013
#1. 'test_prob': 0.01

Best parameter values:
#0. 'beta':      0.018903570702701943
#1. 'test_prob': 0.09536381005175404

Mismatch before calibration: 141.953
Mismatch after calibration:  51.3584
Percent improvement:         63.8%