T7 - Calibration#
We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.
Click here to open an interactive version of this notebook.
The Fit object#
The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:
[1]:
import covasim as cv
cv.options(jupyter=True, verbose=0)
pars = dict(
pop_size = 20_000,
start_day = '2020-02-01',
end_day = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
Covasim 3.1.6 (2024-01-28) — © 2020-2024 by IDM
We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?
[2]:
fit = sim.compute_fit()
fit.summarize()
Mismatch values for:
#0. 'cum_deaths': 55.65217391304348
#1. 'cum_tests': 0.0
#2. 'cum_diagnoses': 22.428498467824312
Total mismatch value:
78.08067238086778
So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)
What if we improve the fit? Does the mismatch reduce?
[3]:
sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim
# Rerun and compute fit
sim.run()
fit = sim.compute_fit()
# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
[4]:
fit.plot()
fit.summarize()
Mismatch values for:
#0. 'cum_deaths': 43.913043478260875
#1. 'cum_tests': 0.0
#2. 'cum_diagnoses': 23.240551583248212
Total mismatch value:
67.1535950615091
As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:
\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)
where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).
Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.
Calibration approaches#
Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).
In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.
However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:
import covasim as cv
import numpy as np
from scipy import optimize
def objective(x, n_runs=10):
print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
pars = dict(
pop_size = 20_000,
start_day = '2020-02-01',
end_day = '2020-04-11',
beta = x[0],
rel_death_prob = x[1],
verbose = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
msim = cv.MultiSim(sim)
msim.run(n_runs=n_runs)
mismatches = []
for sim in msim.sims:
fit = sim.compute_fit()
mismatches.append(fit.mismatch)
mismatch = np.mean(mismatches)
return mismatch
guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization
This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.
What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.
Built-in calibration#
One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things
are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!
[5]:
'''
Example for running built-in calibration with Optuna
'''
import sciris as sc
import covasim as cv
# Create default simulation
pars = sc.objdict(
pop_size = 20_000,
start_day = '2020-02-01',
end_day = '2020-04-11',
beta = 0.015,
rel_death_prob = 1.0,
interventions = cv.test_num(daily_tests='data'),
verbose = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')
# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
beta = [pars.beta, 0.005, 0.20],
rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)
if __name__ == '__main__':
# Run the calibration
calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)
/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-covasim/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db
[I 2025-07-01 05:41:16,013] A new study created in RDB with name: covasim_calibration
[I 2025-07-01 05:41:16,967] Trial 0 finished with value: 1544.3064129324512 and parameters: {'beta': 0.15522297468946203, 'rel_death_prob': 1.075579291447221}. Best is trial 0 with value: 1544.3064129324512.
[I 2025-07-01 05:41:17,031] Trial 1 finished with value: 880.8633476928543 and parameters: {'beta': 0.17325442144782294, 'rel_death_prob': 0.5766370494435211}. Best is trial 1 with value: 880.8633476928543.
[I 2025-07-01 05:41:17,506] Trial 3 finished with value: 133.3430297108851 and parameters: {'beta': 0.0070958813463524815, 'rel_death_prob': 0.7784945435109948}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:17,744] Trial 2 finished with value: 2558.959452857841 and parameters: {'beta': 0.18166378315717352, 'rel_death_prob': 2.4187330429541065}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:18,231] Trial 5 finished with value: 144.77694630723454 and parameters: {'beta': 0.008156199010467489, 'rel_death_prob': 1.9129116869932736}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:18,300] Trial 4 finished with value: 1953.6801083625705 and parameters: {'beta': 0.16944336729375317, 'rel_death_prob': 1.4259407912021538}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:18,999] Trial 6 finished with value: 868.8504241239953 and parameters: {'beta': 0.14283772681406343, 'rel_death_prob': 0.7502960999605333}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:19,067] Trial 7 finished with value: 1166.5218501576587 and parameters: {'beta': 0.12079959300571443, 'rel_death_prob': 1.024063787483286}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:19,473] Trial 8 finished with value: 141.73857529866322 and parameters: {'beta': 0.008138194987831825, 'rel_death_prob': 2.745115946862453}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:19,760] Trial 9 finished with value: 1305.75864902074 and parameters: {'beta': 0.03848123164218227, 'rel_death_prob': 2.2787709404399026}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:20,218] Trial 10 finished with value: 1220.7600923746502 and parameters: {'beta': 0.10158684561163944, 'rel_death_prob': 1.1770145406449908}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:20,490] Trial 11 finished with value: 1537.4186170448995 and parameters: {'beta': 0.07601724062085345, 'rel_death_prob': 1.6375636412402215}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:20,933] Trial 12 finished with value: 1874.3637029799706 and parameters: {'beta': 0.055063880068223714, 'rel_death_prob': 2.875740185526004}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:21,004] Trial 13 finished with value: 138.51978505129458 and parameters: {'beta': 0.008568657482068566, 'rel_death_prob': 2.7880557711447556}. Best is trial 3 with value: 133.3430297108851.
[I 2025-07-01 05:41:21,439] Trial 14 finished with value: 58.83399209486166 and parameters: {'beta': 0.011617010708868964, 'rel_death_prob': 2.8512770428110565}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:21,707] Trial 15 finished with value: 1240.8736732246746 and parameters: {'beta': 0.03556563775885875, 'rel_death_prob': 2.003134239477765}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:22,148] Trial 16 finished with value: 1190.3435182306703 and parameters: {'beta': 0.044818520947656634, 'rel_death_prob': 1.9739201367780357}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:22,439] Trial 17 finished with value: 1960.1823067016032 and parameters: {'beta': 0.07397090522210947, 'rel_death_prob': 2.337450543334615}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:22,878] Trial 18 finished with value: 1698.9899853444063 and parameters: {'beta': 0.07131913059331813, 'rel_death_prob': 2.411495273934409}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:23,127] Trial 19 finished with value: 809.336723364569 and parameters: {'beta': 0.028530991812006808, 'rel_death_prob': 1.403756192534593}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:23,561] Trial 20 finished with value: 789.5688812896922 and parameters: {'beta': 0.02857504807212308, 'rel_death_prob': 1.492682114728328}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:23,877] Trial 21 finished with value: 919.802371541502 and parameters: {'beta': 0.09941943202019836, 'rel_death_prob': 0.8437031409983327}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:24,043] Trial 22 finished with value: 139.8693209574988 and parameters: {'beta': 0.006961159965273679, 'rel_death_prob': 2.6577653300847635}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:24,405] Trial 23 finished with value: 60.5833370342408 and parameters: {'beta': 0.013336989744574358, 'rel_death_prob': 2.967893189168457}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:24,684] Trial 24 finished with value: 411.93453834880313 and parameters: {'beta': 0.020814174569845946, 'rel_death_prob': 2.7055142395360003}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:25,063] Trial 25 finished with value: 643.5812941333215 and parameters: {'beta': 0.022906677074695594, 'rel_death_prob': 2.917757280504762}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:25,410] Trial 26 finished with value: 1585.6551716480883 and parameters: {'beta': 0.056387411112580096, 'rel_death_prob': 2.5601881082832305}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:25,784] Trial 27 finished with value: 1726.9456410711907 and parameters: {'beta': 0.05456097097817799, 'rel_death_prob': 2.5277822818113114}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:26,118] Trial 28 finished with value: 1558.0080605764533 and parameters: {'beta': 0.04636214225422339, 'rel_death_prob': 2.9426905042265252}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:26,587] Trial 29 finished with value: 2556.5200737220766 and parameters: {'beta': 0.1966797028023925, 'rel_death_prob': 2.1695395387628844}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:26,739] Trial 30 finished with value: 287.278278633921 and parameters: {'beta': 0.019539614559252273, 'rel_death_prob': 2.088977505591584}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:27,232] Trial 32 finished with value: 137.92201447795 and parameters: {'beta': 0.0064338724887642515, 'rel_death_prob': 2.9961145208235553}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:27,346] Trial 31 finished with value: 1407.6555491406493 and parameters: {'beta': 0.1223002308726951, 'rel_death_prob': 1.175614441791345}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:27,840] Trial 33 finished with value: 205.57090198516676 and parameters: {'beta': 0.01846531854142042, 'rel_death_prob': 0.5256770119662235}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:27,915] Trial 34 finished with value: 105.86290358395878 and parameters: {'beta': 0.016633536363538995, 'rel_death_prob': 2.9965455869053677}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:28,298] Trial 35 finished with value: 134.90129679797485 and parameters: {'beta': 0.005022707154567017, 'rel_death_prob': 2.9888355476450177}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:28,629] Trial 36 finished with value: 1167.198339032731 and parameters: {'beta': 0.035964129537374635, 'rel_death_prob': 2.544206618354464}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:29,012] Trial 37 finished with value: 1026.647755029533 and parameters: {'beta': 0.03620156326546469, 'rel_death_prob': 1.7952413353795367}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:29,269] Trial 38 finished with value: 330.4426655415908 and parameters: {'beta': 0.019448721078385724, 'rel_death_prob': 1.7598677399060445}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:29,744] Trial 39 finished with value: 1901.4447972642893 and parameters: {'beta': 0.0644806460168874, 'rel_death_prob': 2.778437294549951}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:30,019] Trial 40 finished with value: 565.8448061464671 and parameters: {'beta': 0.08725356331616355, 'rel_death_prob': 0.7212731922760758}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:30,533] Trial 41 finished with value: 2366.480437003153 and parameters: {'beta': 0.1430011155223602, 'rel_death_prob': 2.60710110279375}. Best is trial 14 with value: 58.83399209486166.
[I 2025-07-01 05:41:30,594] Trial 42 finished with value: 36.87513878402984 and parameters: {'beta': 0.013804695977971286, 'rel_death_prob': 2.8859680048693033}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:31,075] Trial 43 finished with value: 38.974996669183284 and parameters: {'beta': 0.013758934092419788, 'rel_death_prob': 2.861563956925799}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:31,148] Trial 44 finished with value: 66.31833725629525 and parameters: {'beta': 0.013731945070161278, 'rel_death_prob': 2.819972249121692}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:31,792] Trial 45 finished with value: 1635.3773815339523 and parameters: {'beta': 0.04524677268430928, 'rel_death_prob': 2.8227416053057395}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:31,866] Trial 46 finished with value: 1158.391104498823 and parameters: {'beta': 0.029046508925166914, 'rel_death_prob': 2.8149902745553876}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:32,467] Trial 47 finished with value: 966.7898698760937 and parameters: {'beta': 0.026474714314557862, 'rel_death_prob': 2.856177852736246}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:32,561] Trial 48 finished with value: 1230.1250166540835 and parameters: {'beta': 0.030256723357117276, 'rel_death_prob': 2.458494119229485}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:32,983] Trial 49 finished with value: 81.35375494071147 and parameters: {'beta': 0.014167790243647962, 'rel_death_prob': 2.4698187441240336}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:33,085] Trial 50 finished with value: 74.54056934760403 and parameters: {'beta': 0.014471862179086059, 'rel_death_prob': 2.656853590160096}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:33,523] Trial 51 finished with value: 57.941999378247544 and parameters: {'beta': 0.013485330040649258, 'rel_death_prob': 2.7073430720680354}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:33,633] Trial 52 finished with value: 66.35919527468135 and parameters: {'beta': 0.01371129159806631, 'rel_death_prob': 2.6995457145248998}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:34,032] Trial 53 finished with value: 103.11409157525426 and parameters: {'beta': 0.01217965142062069, 'rel_death_prob': 2.7574770282999674}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:34,355] Trial 54 finished with value: 1400.4082471021893 and parameters: {'beta': 0.04091340021804536, 'rel_death_prob': 2.7455221876689917}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:34,744] Trial 55 finished with value: 1075.8933250433006 and parameters: {'beta': 0.03595990703293757, 'rel_death_prob': 2.2702186764646495}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:35,069] Trial 56 finished with value: 1759.4270995248037 and parameters: {'beta': 0.05053563585387824, 'rel_death_prob': 2.8939732969303518}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:35,471] Trial 57 finished with value: 1719.7649553670562 and parameters: {'beta': 0.050413898957243373, 'rel_death_prob': 2.8894503982208195}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:35,748] Trial 58 finished with value: 741.5700359728205 and parameters: {'beta': 0.023936239259222843, 'rel_death_prob': 2.3737221743569314}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:36,139] Trial 59 finished with value: 728.8294399786828 and parameters: {'beta': 0.023215451139393287, 'rel_death_prob': 2.6055787299933364}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:36,483] Trial 60 finished with value: 1965.5052848958562 and parameters: {'beta': 0.06334752622490075, 'rel_death_prob': 2.5886715401577094}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:36,913] Trial 61 finished with value: 1872.5687258515788 and parameters: {'beta': 0.06408308462262556, 'rel_death_prob': 2.664260683156834}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:37,007] Trial 62 finished with value: 103.90127459253009 and parameters: {'beta': 0.012148677356741211, 'rel_death_prob': 2.698316935583988}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:37,438] Trial 63 finished with value: 105.71834613847315 and parameters: {'beta': 0.01256411432703412, 'rel_death_prob': 2.711778436389512}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:37,709] Trial 64 finished with value: 1162.1166229959586 and parameters: {'beta': 0.030431395445788236, 'rel_death_prob': 2.8605646679455585}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:38,134] Trial 65 finished with value: 1214.0735888439845 and parameters: {'beta': 0.031097027946749383, 'rel_death_prob': 2.8542636640517394}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:38,174] Trial 66 finished with value: 141.24050717235866 and parameters: {'beta': 0.005100428683799138, 'rel_death_prob': 2.930316963339353}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:38,643] Trial 67 finished with value: 141.52995514500157 and parameters: {'beta': 0.009194784078605845, 'rel_death_prob': 2.9444156245792352}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:38,809] Trial 68 finished with value: 307.4742416840609 and parameters: {'beta': 0.018690954368051977, 'rel_death_prob': 2.490862502592544}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:39,266] Trial 69 finished with value: 312.9902962206333 and parameters: {'beta': 0.019539187020944208, 'rel_death_prob': 2.500592084968509}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:39,578] Trial 70 finished with value: 2373.840875782742 and parameters: {'beta': 0.11956784980002043, 'rel_death_prob': 2.790249859143881}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:39,973] Trial 71 finished with value: 1296.9116667406847 and parameters: {'beta': 0.04132066903500333, 'rel_death_prob': 2.2614981173868864}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:40,100] Trial 72 finished with value: 79.86077186126039 and parameters: {'beta': 0.012711806133068026, 'rel_death_prob': 2.6347890297896233}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:40,493] Trial 73 finished with value: 72.83763378780478 and parameters: {'beta': 0.014464217495669184, 'rel_death_prob': 2.6398929707437384}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:40,780] Trial 74 finished with value: 798.0623528889283 and parameters: {'beta': 0.024306690359782883, 'rel_death_prob': 2.6858988658463545}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:41,163] Trial 75 finished with value: 859.063263312164 and parameters: {'beta': 0.024342627697642248, 'rel_death_prob': 2.769482851178232}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:41,349] Trial 76 finished with value: 97.28627259404004 and parameters: {'beta': 0.01607432043423383, 'rel_death_prob': 2.764010494750891}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:41,629] Trial 77 finished with value: 141.3400985921748 and parameters: {'beta': 0.0053006311943218035, 'rel_death_prob': 2.997447665265272}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:41,854] Trial 78 finished with value: 123.86430252697961 and parameters: {'beta': 0.008951039890581033, 'rel_death_prob': 2.9443467760113404}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:42,332] Trial 79 finished with value: 1224.6005240484967 and parameters: {'beta': 0.03269618969848204, 'rel_death_prob': 2.913667136205295}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:42,645] Trial 80 finished with value: 2261.403939245903 and parameters: {'beta': 0.1664438778873255, 'rel_death_prob': 2.3968120674063917}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:43,130] Trial 81 finished with value: 2001.9560776302349 and parameters: {'beta': 0.17248960434517588, 'rel_death_prob': 1.5795362609220223}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:43,200] Trial 82 finished with value: 75.00188746280588 and parameters: {'beta': 0.015455220150817186, 'rel_death_prob': 2.649159727903368}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:43,677] Trial 83 finished with value: 49.53301949638052 and parameters: {'beta': 0.015748162603861124, 'rel_death_prob': 2.6488106106257105}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:43,731] Trial 84 finished with value: 85.96871252831194 and parameters: {'beta': 0.010864019342521423, 'rel_death_prob': 2.823321562587446}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:44,313] Trial 85 finished with value: 224.53868188479817 and parameters: {'beta': 0.019689952145544035, 'rel_death_prob': 2.830587268297131}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:44,394] Trial 86 finished with value: 416.3006839276991 and parameters: {'beta': 0.020957724673243965, 'rel_death_prob': 2.534318468586712}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:45,015] Trial 87 finished with value: 925.9673802016255 and parameters: {'beta': 0.02733711890630227, 'rel_death_prob': 2.543868891020768}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:45,099] Trial 88 finished with value: 1050.101256828174 and parameters: {'beta': 0.027686203156303844, 'rel_death_prob': 2.719871395110057}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:45,770] Trial 89 finished with value: 1997.8489585646398 and parameters: {'beta': 0.0867057585420017, 'rel_death_prob': 2.732756181407661}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:45,838] Trial 90 finished with value: 1303.4100901541058 and parameters: {'beta': 0.03404355523952609, 'rel_death_prob': 2.602709137813382}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:46,386] Trial 92 finished with value: 54.270218057467694 and parameters: {'beta': 0.015294543199397234, 'rel_death_prob': 2.6787663060166085}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:46,468] Trial 91 finished with value: 1445.3058800017766 and parameters: {'beta': 0.0331870734335688, 'rel_death_prob': 2.5997513576793696}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:46,882] Trial 93 finished with value: 141.52995514500157 and parameters: {'beta': 0.009168059369939786, 'rel_death_prob': 2.883963856548719}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:47,013] Trial 94 finished with value: 50.26979615401696 and parameters: {'beta': 0.015780502807956523, 'rel_death_prob': 2.8825301582856717}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:47,457] Trial 95 finished with value: 151.89845450104366 and parameters: {'beta': 0.01615994949591473, 'rel_death_prob': 2.808215250226957}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:47,721] Trial 96 finished with value: 1583.989430208287 and parameters: {'beta': 0.039117566619883054, 'rel_death_prob': 2.967755482801636}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:48,166] Trial 97 finished with value: 1094.5909757072434 and parameters: {'beta': 0.03965705546570454, 'rel_death_prob': 1.8674042742089094}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:48,388] Trial 98 finished with value: 522.6260158990984 and parameters: {'beta': 0.022315477982086017, 'rel_death_prob': 1.3139618415004555}. Best is trial 42 with value: 36.87513878402984.
[I 2025-07-01 05:41:48,706] Trial 99 finished with value: 584.1267708842208 and parameters: {'beta': 0.02359062333756904, 'rel_death_prob': 1.370850202787999}. Best is trial 42 with value: 36.87513878402984.
Making results structure...
Processed 100 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 100 total trials completed in 33.6 s.
Initial parameter values:
#0. 'beta': 0.015
#1. 'rel_death_prob': 1.0
Best parameter values:
#0. 'beta': 0.013804695977971286
#1. 'rel_death_prob': 2.8859680048693033
Mismatch before calibration: 101.465
Mismatch after calibration: 36.8751
Percent improvement: 63.7%
So it improved the fit (see above), but let’s visualize this as a plot:
[6]:
# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.
Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.
[7]:
pars = dict(
verbose = 0,
start_day = '2020-02-05',
pop_size = 1000,
pop_scale = 4,
interventions = cv.test_prob(symp_prob=0.01),
)
sim = cv.Sim(pars, datafile='example_data.csv')
calib_pars = dict(
beta = [0.013, 0.005, 0.020],
test_prob = [0.01, 0.00, 0.30]
)
def set_test_prob(sim, calib_pars):
tp = sim.get_intervention(cv.test_prob)
tp.symp_prob = calib_pars['test_prob']
return sim
calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])
[I 2025-07-01 05:41:50,947] A new study created in RDB with name: covasim_calibration
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db
[I 2025-07-01 05:41:51,312] Trial 0 finished with value: 70.75608561138716 and parameters: {'beta': 0.013795498400840142, 'test_prob': 0.17394988717203244}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:51,351] Trial 1 finished with value: 140.87814778485392 and parameters: {'beta': 0.012955717713710613, 'test_prob': 0.04131035035771412}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:51,536] Trial 2 finished with value: 76.41263820477067 and parameters: {'beta': 0.01840039910209604, 'test_prob': 0.07385737110506081}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:51,613] Trial 3 finished with value: 109.15705451080522 and parameters: {'beta': 0.013731676351352043, 'test_prob': 0.07347292679293117}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:51,747] Trial 4 finished with value: 88.5690256385061 and parameters: {'beta': 0.016375386462613266, 'test_prob': 0.08670209624909068}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:51,823] Trial 5 finished with value: 132.476115386566 and parameters: {'beta': 0.013307533682961094, 'test_prob': 0.24837884941314026}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:51,959] Trial 6 finished with value: 122.03529152792746 and parameters: {'beta': 0.015256196653257885, 'test_prob': 0.19328797581768287}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,039] Trial 7 finished with value: 133.89027073788333 and parameters: {'beta': 0.01630259160196846, 'test_prob': 0.040537517350468774}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,171] Trial 8 finished with value: 128.14948761495353 and parameters: {'beta': 0.01335132066941902, 'test_prob': 0.16113156067988127}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,252] Trial 9 finished with value: 127.66983326399759 and parameters: {'beta': 0.011210662084721917, 'test_prob': 0.02417952933640973}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,391] Trial 10 finished with value: 119.82203453765796 and parameters: {'beta': 0.013336790527193276, 'test_prob': 0.14018439173155506}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,468] Trial 11 finished with value: 144.64311480767452 and parameters: {'beta': 0.007173200346521646, 'test_prob': 0.2749274324261143}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,617] Trial 12 finished with value: 90.70691330299361 and parameters: {'beta': 0.01934610377089565, 'test_prob': 0.2801799009638272}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,694] Trial 13 finished with value: 112.01264870364565 and parameters: {'beta': 0.019368388790998958, 'test_prob': 0.11091240557024731}. Best is trial 0 with value: 70.75608561138716.
[I 2025-07-01 05:41:52,842] Trial 14 finished with value: 70.06039041406449 and parameters: {'beta': 0.019807854710303704, 'test_prob': 0.10500619819936848}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:52,898] Trial 15 finished with value: 136.07008400632992 and parameters: {'beta': 0.009830044661231288, 'test_prob': 0.20228169306732796}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,042] Trial 16 finished with value: 136.15193784372002 and parameters: {'beta': 0.010160623988261704, 'test_prob': 0.2123470905098795}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,123] Trial 17 finished with value: 145.27075664052637 and parameters: {'beta': 0.005846605628695947, 'test_prob': 0.1355654430966718}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,233] Trial 18 finished with value: 144.82311549036888 and parameters: {'beta': 0.005589942281046438, 'test_prob': 0.14512084331333863}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,353] Trial 19 finished with value: 85.29895941258003 and parameters: {'beta': 0.017508844056220642, 'test_prob': 0.164377217958867}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,468] Trial 20 finished with value: 135.82272072179046 and parameters: {'beta': 0.017854787592970864, 'test_prob': 0.0006037166899153651}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,547] Trial 21 finished with value: 141.05597228576323 and parameters: {'beta': 0.008340132640793974, 'test_prob': 0.23501949697303026}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,694] Trial 22 finished with value: 90.79307474280188 and parameters: {'beta': 0.01970911704836855, 'test_prob': 0.09732574439862399}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,779] Trial 23 finished with value: 95.92949481082393 and parameters: {'beta': 0.01991512374686448, 'test_prob': 0.09835432980348774}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:53,921] Trial 24 finished with value: 119.8716445108422 and parameters: {'beta': 0.01795704206677632, 'test_prob': 0.06344081732366759}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,009] Trial 25 finished with value: 104.59832865100383 and parameters: {'beta': 0.017990251978351458, 'test_prob': 0.06208415412096273}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,147] Trial 26 finished with value: 121.23876316010163 and parameters: {'beta': 0.014994704457387526, 'test_prob': 0.11931579358813305}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,237] Trial 27 finished with value: 98.60471023109699 and parameters: {'beta': 0.01485293216726425, 'test_prob': 0.12120001546209215}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,376] Trial 28 finished with value: 119.81152913543293 and parameters: {'beta': 0.01633968831401317, 'test_prob': 0.1783954083324557}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,462] Trial 29 finished with value: 123.42618644129972 and parameters: {'beta': 0.016271459266712508, 'test_prob': 0.18319254750673494}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,596] Trial 30 finished with value: 139.9671182329635 and parameters: {'beta': 0.011773603221796816, 'test_prob': 0.04082939938802213}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,677] Trial 31 finished with value: 145.01492743893968 and parameters: {'beta': 0.011681152475646954, 'test_prob': 0.037390479773752566}. Best is trial 14 with value: 70.06039041406449.
[I 2025-07-01 05:41:54,823] Trial 32 finished with value: 55.80450566852936 and parameters: {'beta': 0.01845840516688568, 'test_prob': 0.16469354199371325}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:54,907] Trial 33 finished with value: 62.69342021762304 and parameters: {'beta': 0.018729019678869467, 'test_prob': 0.16375248785050894}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,052] Trial 34 finished with value: 83.19618628225672 and parameters: {'beta': 0.01873822554587939, 'test_prob': 0.07812692056760899}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,137] Trial 35 finished with value: 103.44489087475431 and parameters: {'beta': 0.018838034731784144, 'test_prob': 0.22592375026881195}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,277] Trial 36 finished with value: 106.18170241773437 and parameters: {'beta': 0.017160920812301486, 'test_prob': 0.1735008076367397}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,369] Trial 37 finished with value: 77.3307402205328 and parameters: {'beta': 0.017647774036670862, 'test_prob': 0.16056799786272577}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,500] Trial 38 finished with value: 105.67541432637499 and parameters: {'beta': 0.014318615434220954, 'test_prob': 0.1548906267641705}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,602] Trial 39 finished with value: 128.7895990351866 and parameters: {'beta': 0.014207701610638768, 'test_prob': 0.1983506103011582}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,736] Trial 40 finished with value: 107.77982303423306 and parameters: {'beta': 0.018684797333711547, 'test_prob': 0.19373811209846575}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,834] Trial 41 finished with value: 75.4877944467784 and parameters: {'beta': 0.016863343099119566, 'test_prob': 0.26304771619479006}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:55,964] Trial 42 finished with value: 88.50806020487327 and parameters: {'beta': 0.016760052620297863, 'test_prob': 0.12781799243980074}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,063] Trial 43 finished with value: 131.4946618925491 and parameters: {'beta': 0.015799392128739476, 'test_prob': 0.29434309861570784}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,192] Trial 44 finished with value: 117.03765361080316 and parameters: {'beta': 0.01576315131877669, 'test_prob': 0.2606741808966646}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,285] Trial 45 finished with value: 133.4145370641948 and parameters: {'beta': 0.012613623621500356, 'test_prob': 0.2668433068249245}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,428] Trial 46 finished with value: 84.36318163290623 and parameters: {'beta': 0.019042615393651763, 'test_prob': 0.2213508982906123}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,511] Trial 47 finished with value: 66.33568000140954 and parameters: {'beta': 0.019119771485034222, 'test_prob': 0.21822024050066774}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,656] Trial 48 finished with value: 116.55052263888362 and parameters: {'beta': 0.01972060914785334, 'test_prob': 0.24960438930812695}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,740] Trial 49 finished with value: 65.61834185339893 and parameters: {'beta': 0.019945141176812793, 'test_prob': 0.23656671006475197}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,888] Trial 50 finished with value: 57.84534330194478 and parameters: {'beta': 0.01825787488325349, 'test_prob': 0.14137834671051644}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:56,968] Trial 51 finished with value: 98.25595373353897 and parameters: {'beta': 0.01844292051358249, 'test_prob': 0.24145319474805793}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:57,119] Trial 52 finished with value: 85.49353599475836 and parameters: {'beta': 0.019993858925115824, 'test_prob': 0.14412376574843525}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:57,200] Trial 53 finished with value: 77.90607334998192 and parameters: {'beta': 0.0193412086251732, 'test_prob': 0.20939276335777998}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:57,349] Trial 54 finished with value: 99.52658248000512 and parameters: {'beta': 0.019243394888437775, 'test_prob': 0.20696833591208674}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:57,430] Trial 55 finished with value: 87.26322999711623 and parameters: {'beta': 0.018293064042108263, 'test_prob': 0.10598169157705124}. Best is trial 32 with value: 55.80450566852936.
[I 2025-07-01 05:41:57,579] Trial 56 finished with value: 51.16369359129747 and parameters: {'beta': 0.018306921304561987, 'test_prob': 0.10372756620520418}. Best is trial 56 with value: 51.16369359129747.
[I 2025-07-01 05:41:57,661] Trial 57 finished with value: 93.98713149776812 and parameters: {'beta': 0.01736000394940693, 'test_prob': 0.16573286055144804}. Best is trial 56 with value: 51.16369359129747.
[I 2025-07-01 05:41:57,807] Trial 58 finished with value: 83.99396886431612 and parameters: {'beta': 0.017455273541502402, 'test_prob': 0.1897534990095872}. Best is trial 56 with value: 51.16369359129747.
[I 2025-07-01 05:41:57,871] Trial 59 finished with value: 68.77217171069486 and parameters: {'beta': 0.01851666109950188, 'test_prob': 0.13223922678623373}. Best is trial 56 with value: 51.16369359129747.
Making results structure...
Processed 60 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 60 total trials completed in 7.4 s.
Initial parameter values:
#0. 'beta': 0.013
#1. 'test_prob': 0.01
Best parameter values:
#0. 'beta': 0.018306921304561987
#1. 'test_prob': 0.10372756620520418
Mismatch before calibration: 141.953
Mismatch after calibration: 51.1637
Percent improvement: 64.0%
