T7 - Calibration#
An interactive version of this notebook is available on Google Colab or Binder.
Disease models typically require contextualization to a relevant setting of interest prior to addressing “what-if” scenario questions. The process of tuning model input parameters so that model outputs match observed data is known as calibration. There are many approaches to model calibration, ranging from manual tuning to fully Bayesian methods.
For many applications, we have found that an optimization-based approach is sufficient. Such methods avoid the tedious process of manual tuning and are less computationally expensive than fully Bayesian methods. One such optimization-based approach is the Optuna library, which is a Bayesian hyperparameter optimization framework. Optuna is designed for tuning hyperparameters of machine learning models, but it can also be used to calibrate disease models.
Calibration libraries often treat the disease model as a black box, where the input parameters are the “hyperparameters” to be tuned. The calibration process is often iterative and requires a combination of expert knowledge and computational tools. The optimization algorithm iteratively chooses new parameter values to evaluate, and the model is run with these values to generate outputs. The outputs are compared to observed data, and a loss function is calculated to quantify the difference between the model outputs and the observed data. The optimization algorithm then uses this loss function to update its search strategy and choose new parameter values to evaluate. This process continues until the algorithm converges to a set of parameter values that minimize the loss function.
While many optimization algorithms are available, Starsim has a built-in interface to the Optuna library, which we will demonstrate in this tutorial. We will use a simple Susceptible-Infected-Recovered (SIR) model as an example. We will tune three input parameters, the infectivity parameter, beta, the initial prevalence parameter, init_prev, and the Poisson-distributed degree distribution parameter, n_contacts. We will calibrate the model using a beta-binomial likelihood function so
as to match prevalence at three distinct time points.
We begin with a few imports and default settings:
[1]:
#%% Imports and settings
import sciris as sc
import starsim as ss
import pandas as pd
n_agents = 2e3
debug = False # If true, will run in serial
The calibration class will require a base Sim object. This sim will later be modified according to parameters selected by the optimization engine. The following function creates the base Sim object.
[2]:
def make_sim():
""" Helper function to create the base simulation object """
sir = ss.SIR(
beta = ss.beta(0.075),
init_prev = ss.bernoulli(0.02),
)
random = ss.RandomNet(n_contacts=ss.poisson(4))
sim = ss.Sim(
n_agents = n_agents,
start = sc.date('1990-01-01'),
dur = 40,
dt = 1,
unit = 'day',
diseases = sir,
networks = random,
verbose = 0,
)
# Remember to return the sim object
return sim
Now let’s define the calibration parameters. These are the inputs that Optuna will be able to modify. Here, we define three such parameters, beta, init_prev, and n_contacts.
Each parameter entry should have range defined by low and high as well as a guess values. The guess value is not used by Optuna, rather only for a check after calibration completes to see if the new parameters are better than the guess values.
You’ll notice there are a few other parameters that can be specified. For example, the data type of the parameter appears in suggest_type. Possible values are listed in the Optuna documentation, and include suggest_float for float values and
suggest_int for integer types.
To make things easier for the search algorithm, it’s helpful to indicate how outputs are expected to change with inputs. For example, increasing beta from 0.01 to 0.02 should double disease transmission, but increasing from 0.11 to 0.12 will have a small effect. Thus, we indicate that this parameter should be calibrated with log=True.
[3]:
# Define the calibration parameters
calib_pars = dict(
beta = dict(low=0.01, high=0.30, guess=0.15, suggest_type='suggest_float', log=True), # Note the log scale
init_prev = dict(low=0.01, high=0.05, guess=0.15), # Default type is suggest_float, no need to re-specify
n_contacts = dict(low=2, high=10, guess=3, suggest_type='suggest_int'), # Suggest int just for this demo
)
The optimization engine iteratively chooses input parameters to simulate. Those parameters are passed into the following build_sim function as a dictionary of calib_pars along with the base sim and any other key word arguments. The calib_pars will be as above, but importantly will have an additional key named value containing the value selected by Optuna.
When modifying a sim, it is important to realize that the simulation has not been initialized yet. Nonetheless, the configuration is available for modification at sim.pars, as demonstrated in the function below for the SIR example.
[4]:
def build_sim(sim, calib_pars, **kwargs):
""" Modify the base simulation by applying calib_pars """
sir = sim.pars.diseases # There is only one disease in this simulation and it is a SIR
net = sim.pars.networks # There is only one network in this simulation and it is a RandomNet
for k, pars in calib_pars.items(): # Loop over the calibration parameters
if k == 'rand_seed':
sim.pars.rand_seed = v
continue
# Each item in calib_pars is a dictionary with keys like 'low', 'high',
# 'guess', 'suggest_type', and importantly 'value'. The 'value' key is
# the one we want to use as that's the one selected by the algorithm
v = pars['value']
if k == 'beta':
sir.pars.beta = ss.beta(v)
elif k == 'init_prev':
sir.pars.init_prev = ss.bernoulli(v)
elif k == 'n_contacts':
net.pars.n_contacts = ss.poisson(v)
else:
raise NotImplementedError(f'Parameter {k} not recognized')
return sim
The Starsim framework has been integrated with the Optuna hyperparameter optimization algorithm to facilitate calibration through the Calibration class. Recall that an optimization-based approach to calibration minimizes a function of the input parameters. This function is key to achieving an acceptable calibration.
There are two ways to describe the goodness-of-fit function for the Calibration. The first method is to directly provide a function that the algorithm will call. The eval_fn will be passed each completed sim after running, and is expected to return a float representing the goodness of fit (higher is better). Data can be passed into the eval_fn via eval_kwargs.
As an alternative to directly specifying the evaluation function, you can use CalibComponents. Each component includes real data, for example from a survey, that is compared against simulation data from the model. Several components and be used at the same time, for example one for disease prevalence and another for treatment coverage. Each component computes a likelihood of the data given the input parameters, as assess via simulation. Components are combined assuming independence.
When defining a CalibComponent, we give it a name and pass in expected (the real data to be calibrated to). The required data fields depend on the likelihood function. Importantly, the functional form of the negative log likelihood, or nll, is defined by the nll_fn. The value for nll_fn can be 'beta', 'gamma', or a negative log likelihood function of your own creation. If designing your own function for nll_fn, it should take two arguments: expected and
actual. For a beta binomial, the data must define n and x, where n is the number of individuals who were sampled and x is the number that were found, e.g. identified as positive.
Output from the simulation is obtained via a function. The function takes a completed sim object as input and returns a dictionary with fields as required for the evaluation function of your choice. In the example below, we use an in-line lambda function to extract n and x from the simulation, as required by the Beta binomial component.
Each component has a weight. The final goodness of fit is a weighted sum of negative log likelihoods.
Finally, the conform argument describes how the simulation output is adjusted to align with the real data. For example, if the real data is a prevalence measurement, choosing 'prevalent' will interpolate the simulation output at the time points of the real data. Choosing 'incident', the simulation output will be aggregated between time points of the real data.
[5]:
infectious = ss.CalibComponent(
name = 'Infectious',
# For this example, the "expected" comes from a simulation with pars
# beta=0.075, init_prev=0.02, n_contacts=4
expected = pd.DataFrame({
'n': [200, 197, 195], # Number of individuals sampled
'x': [30, 30, 10], # Number of individuals found to be infectious
}, index=pd.Index([ss.date(d) for d in ['1990-01-12', '1990-01-25', '1990-02-02']], name='t')), # On these dates
extract_fn = lambda sim: pd.DataFrame({
'n': sim.results.n_alive, # Number of individuals sampled
'x': sim.results.sir.n_infected, # Number of individuals found to be infectious
}, index=pd.Index(sim.results.timevec, name='t')), # Index is time
conform = 'prevalent',
nll_fn = 'beta',
weight = 1, # Not required if only one component
)
Finally, we can bring all the pieces together. We make a single base simulation and create an instance of a Starsim Calibration object. This object requires a few arguments, like the calib_pars and sim. We also pass in the function that modifies the base sim, here our build_sim function. No additional build_kw are required in this example.
We also pass in a list of components. Instead of using this “component-based” system, a user could simply provide an eval_fn, which takes in a completed sim an any eval_kwargs and returns a “goodness of fit” score to be maximized.
We can also specify the total number of trial to run, the number of parallel works, and a few other parameters.
[6]:
sc.heading('Beginning calibration')
# Make the sim and data
sim = make_sim()
# Make the calibration
calib = ss.Calibration(
calib_pars = calib_pars,
sim = sim,
build_fn = build_sim, # Use default builder, Calibration.translate_pars
build_kw = None,
components = [infectious],
total_trials = 100,
n_workers = None, # None indicates to use all available CPUs
die = True,
debug = debug,
)
# Perform the calibration
sc.printcyan('\nPeforming calibration...')
calib.calibrate();
—————————————————————
Beginning calibration
—————————————————————
Peforming calibration...
Removed existing calibration file starsim_calibration.db
sqlite:///starsim_calibration.db
[I 2024-11-21 04:20:31,759] A new study created in RDB with name: starsim_calibration
[I 2024-11-21 04:20:32,883] Trial 0 finished with value: 145.96510895587448 and parameters: {'beta': 0.01222552552205945, 'init_prev': 0.01763701404210597, 'n_contacts': 7, 'rand_seed': 655543}. Best is trial 0 with value: 145.96510895587448.
[I 2024-11-21 04:20:32,911] Trial 1 finished with value: 169.85989928869344 and parameters: {'beta': 0.011725398610445218, 'init_prev': 0.016135698058387354, 'n_contacts': 2, 'rand_seed': 524144}. Best is trial 0 with value: 145.96510895587448.
[I 2024-11-21 04:20:33,266] Trial 2 finished with value: 120.60306650319649 and parameters: {'beta': 0.2377334094386809, 'init_prev': 0.03846637388298969, 'n_contacts': 9, 'rand_seed': 595871}. Best is trial 2 with value: 120.60306650319649.
[I 2024-11-21 04:20:33,368] Trial 3 finished with value: 147.90939467565636 and parameters: {'beta': 0.07194369653578059, 'init_prev': 0.030176696874261687, 'n_contacts': 8, 'rand_seed': 944103}. Best is trial 2 with value: 120.60306650319649.
[I 2024-11-21 04:20:33,668] Trial 4 finished with value: 112.36786492336319 and parameters: {'beta': 0.16259919360951447, 'init_prev': 0.04949670296802118, 'n_contacts': 10, 'rand_seed': 697900}. Best is trial 4 with value: 112.36786492336319.
[I 2024-11-21 04:20:33,754] Trial 5 finished with value: 120.1024015452756 and parameters: {'beta': 0.05687104882423437, 'init_prev': 0.03300599090586321, 'n_contacts': 2, 'rand_seed': 387674}. Best is trial 4 with value: 112.36786492336319.
[I 2024-11-21 04:20:34,018] Trial 6 finished with value: 167.16892974277903 and parameters: {'beta': 0.014706553625199464, 'init_prev': 0.03723376728631156, 'n_contacts': 2, 'rand_seed': 839085}. Best is trial 4 with value: 112.36786492336319.
[I 2024-11-21 04:20:34,154] Trial 7 finished with value: 44.292759134993275 and parameters: {'beta': 0.04426029156197891, 'init_prev': 0.047826559277586375, 'n_contacts': 4, 'rand_seed': 885693}. Best is trial 7 with value: 44.292759134993275.
[I 2024-11-21 04:20:34,415] Trial 8 finished with value: 89.87511165977298 and parameters: {'beta': 0.03438093683092403, 'init_prev': 0.02412668522684437, 'n_contacts': 4, 'rand_seed': 658453}. Best is trial 7 with value: 44.292759134993275.
[I 2024-11-21 04:20:34,564] Trial 9 finished with value: 111.19926312200982 and parameters: {'beta': 0.18524843913242978, 'init_prev': 0.030881704119983942, 'n_contacts': 10, 'rand_seed': 922796}. Best is trial 7 with value: 44.292759134993275.
[I 2024-11-21 04:20:34,835] Trial 10 finished with value: 42.159442539154725 and parameters: {'beta': 0.023458124083620517, 'init_prev': 0.039509668883204875, 'n_contacts': 7, 'rand_seed': 938445}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:35,004] Trial 11 finished with value: 74.05583965064284 and parameters: {'beta': 0.030597140337240186, 'init_prev': 0.04856885112581281, 'n_contacts': 5, 'rand_seed': 196485}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:35,267] Trial 12 finished with value: 80.89058518908746 and parameters: {'beta': 0.02784420066669349, 'init_prev': 0.048761128489152504, 'n_contacts': 5, 'rand_seed': 38570}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:35,444] Trial 13 finished with value: 70.04401019129978 and parameters: {'beta': 0.026194445452952655, 'init_prev': 0.04307584515923341, 'n_contacts': 6, 'rand_seed': 804153}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:35,683] Trial 14 finished with value: 138.0157631944728 and parameters: {'beta': 0.09256654401863203, 'init_prev': 0.042387066309922554, 'n_contacts': 6, 'rand_seed': 998637}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:35,900] Trial 15 finished with value: 55.239385934037045 and parameters: {'beta': 0.10088690996161877, 'init_prev': 0.04205087773337407, 'n_contacts': 4, 'rand_seed': 982543}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:36,094] Trial 16 finished with value: 145.03533145270626 and parameters: {'beta': 0.01980184059524009, 'init_prev': 0.04169369876312915, 'n_contacts': 4, 'rand_seed': 385068}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:36,342] Trial 17 finished with value: 109.07152226994617 and parameters: {'beta': 0.0193144930182887, 'init_prev': 0.02465911203796757, 'n_contacts': 8, 'rand_seed': 393806}. Best is trial 10 with value: 42.159442539154725.
[I 2024-11-21 04:20:36,553] Trial 18 finished with value: 22.587844431935764 and parameters: {'beta': 0.0448138137527222, 'init_prev': 0.023674076356472243, 'n_contacts': 8, 'rand_seed': 819224}. Best is trial 18 with value: 22.587844431935764.
[I 2024-11-21 04:20:36,797] Trial 19 finished with value: 11.034789500056604 and parameters: {'beta': 0.04381385689240407, 'init_prev': 0.03641816077495649, 'n_contacts': 7, 'rand_seed': 798158}. Best is trial 19 with value: 11.034789500056604.
[I 2024-11-21 04:20:37,008] Trial 20 finished with value: 23.11201307983788 and parameters: {'beta': 0.04788908427693344, 'init_prev': 0.01024197716620829, 'n_contacts': 8, 'rand_seed': 761616}. Best is trial 19 with value: 11.034789500056604.
[I 2024-11-21 04:20:37,254] Trial 21 finished with value: 11.685379089015214 and parameters: {'beta': 0.04638169427560693, 'init_prev': 0.023932383621332, 'n_contacts': 8, 'rand_seed': 755813}. Best is trial 19 with value: 11.034789500056604.
[I 2024-11-21 04:20:37,469] Trial 22 finished with value: 9.453603103673913 and parameters: {'beta': 0.04611275025137568, 'init_prev': 0.01205613477454643, 'n_contacts': 8, 'rand_seed': 758001}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:37,706] Trial 23 finished with value: 155.97032047735206 and parameters: {'beta': 0.0726890231163556, 'init_prev': 0.02590710295344306, 'n_contacts': 9, 'rand_seed': 736936}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:37,920] Trial 24 finished with value: 173.6796380532212 and parameters: {'beta': 0.06732599778088946, 'init_prev': 0.012850786281311354, 'n_contacts': 9, 'rand_seed': 719356}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:38,162] Trial 25 finished with value: 15.659026932181177 and parameters: {'beta': 0.037027447479426774, 'init_prev': 0.01047494932259567, 'n_contacts': 7, 'rand_seed': 504538}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:38,376] Trial 26 finished with value: 9.88444813556157 and parameters: {'beta': 0.037131958242317284, 'init_prev': 0.020416618557897816, 'n_contacts': 7, 'rand_seed': 536260}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:38,592] Trial 27 finished with value: 181.48565620832483 and parameters: {'beta': 0.13378621838669277, 'init_prev': 0.01765894049959541, 'n_contacts': 6, 'rand_seed': 578022}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:38,811] Trial 28 finished with value: 172.5014448012505 and parameters: {'beta': 0.10877049875075838, 'init_prev': 0.018661415861601305, 'n_contacts': 6, 'rand_seed': 585862}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:39,031] Trial 29 finished with value: 180.656856973839 and parameters: {'beta': 0.10001078929182043, 'init_prev': 0.021158939811645003, 'n_contacts': 7, 'rand_seed': 311093}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:39,257] Trial 30 finished with value: 61.25351115895296 and parameters: {'beta': 0.05788448906453835, 'init_prev': 0.020703372101796458, 'n_contacts': 7, 'rand_seed': 433647}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:39,492] Trial 31 finished with value: 54.52455218057253 and parameters: {'beta': 0.06143188447731179, 'init_prev': 0.01481001700456799, 'n_contacts': 7, 'rand_seed': 624043}. Best is trial 22 with value: 9.453603103673913.
[I 2024-11-21 04:20:39,720] Trial 32 finished with value: 7.544700950781362 and parameters: {'beta': 0.03798904967029662, 'init_prev': 0.0275687855655205, 'n_contacts': 8, 'rand_seed': 647595}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:39,955] Trial 33 finished with value: 18.72443375015314 and parameters: {'beta': 0.03998273556842251, 'init_prev': 0.027215013799264606, 'n_contacts': 8, 'rand_seed': 673464}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:40,191] Trial 34 finished with value: 18.21917247439046 and parameters: {'beta': 0.03471661686491395, 'init_prev': 0.034873421669672194, 'n_contacts': 9, 'rand_seed': 667676}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:40,421] Trial 35 finished with value: 11.17354975628291 and parameters: {'beta': 0.031549803401135, 'init_prev': 0.03523426779836115, 'n_contacts': 9, 'rand_seed': 526473}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:40,658] Trial 36 finished with value: 45.66102123858104 and parameters: {'beta': 0.02130316410406258, 'init_prev': 0.029093952404864978, 'n_contacts': 9, 'rand_seed': 546050}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:40,843] Trial 37 finished with value: 152.21995434821542 and parameters: {'beta': 0.014748071363335653, 'init_prev': 0.028131969880359077, 'n_contacts': 5, 'rand_seed': 872107}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:41,071] Trial 38 finished with value: 159.2201680414771 and parameters: {'beta': 0.013769650223939106, 'init_prev': 0.014662311894363227, 'n_contacts': 5, 'rand_seed': 463974}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:41,300] Trial 39 finished with value: 148.85295064151137 and parameters: {'beta': 0.010527464573600178, 'init_prev': 0.015223648298997, 'n_contacts': 10, 'rand_seed': 794592}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:41,523] Trial 40 finished with value: 142.87353088922805 and parameters: {'beta': 0.08469357802810834, 'init_prev': 0.033260480308344026, 'n_contacts': 7, 'rand_seed': 625482}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:41,935] Trial 41 finished with value: 49.49397181944619 and parameters: {'beta': 0.05378605413834599, 'init_prev': 0.031461987903968244, 'n_contacts': 7, 'rand_seed': 613383}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:41,983] Trial 42 finished with value: 93.30728712149062 and parameters: {'beta': 0.05002100512597526, 'init_prev': 0.03669416453294628, 'n_contacts': 9, 'rand_seed': 527552}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:42,376] Trial 43 finished with value: 8.954108217615385 and parameters: {'beta': 0.030911156393927178, 'init_prev': 0.03614153149983556, 'n_contacts': 8, 'rand_seed': 498091}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:42,649] Trial 44 finished with value: 21.26293239547772 and parameters: {'beta': 0.028315269283066675, 'init_prev': 0.0349344690533061, 'n_contacts': 8, 'rand_seed': 298358}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:42,846] Trial 45 finished with value: 32.795599208910744 and parameters: {'beta': 0.02551756422721129, 'init_prev': 0.03825780640962107, 'n_contacts': 8, 'rand_seed': 323268}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:43,116] Trial 46 finished with value: 32.71173384223471 and parameters: {'beta': 0.041472420094351176, 'init_prev': 0.040137982363105666, 'n_contacts': 8, 'rand_seed': 704907}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:43,273] Trial 47 finished with value: 110.93542165639587 and parameters: {'beta': 0.2911623411044909, 'init_prev': 0.04480119747534027, 'n_contacts': 6, 'rand_seed': 879140}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:43,569] Trial 48 finished with value: 44.39470605337988 and parameters: {'beta': 0.03335361080714379, 'init_prev': 0.02070179209940326, 'n_contacts': 6, 'rand_seed': 869510}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:43,738] Trial 49 finished with value: 17.7783371656667 and parameters: {'beta': 0.03438930582364259, 'init_prev': 0.03131746899263078, 'n_contacts': 7, 'rand_seed': 461739}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:44,014] Trial 50 finished with value: 102.68974555335149 and parameters: {'beta': 0.017400606397840707, 'init_prev': 0.03149919922623219, 'n_contacts': 7, 'rand_seed': 461653}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:44,173] Trial 51 finished with value: 133.87326378798093 and parameters: {'beta': 0.022767988347274972, 'init_prev': 0.0122392299874295, 'n_contacts': 7, 'rand_seed': 782008}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:44,483] Trial 52 finished with value: 26.343216800853725 and parameters: {'beta': 0.02364955861907683, 'init_prev': 0.03592846554712376, 'n_contacts': 9, 'rand_seed': 564342}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:44,644] Trial 53 finished with value: 14.160122170251952 and parameters: {'beta': 0.030209523950251567, 'init_prev': 0.03611127416514074, 'n_contacts': 9, 'rand_seed': 572833}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:44,957] Trial 54 finished with value: 39.461415550282936 and parameters: {'beta': 0.03925757566274437, 'init_prev': 0.03385862966811212, 'n_contacts': 10, 'rand_seed': 648898}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:45,128] Trial 55 finished with value: 49.65619266301633 and parameters: {'beta': 0.041503045525764545, 'init_prev': 0.03279601583701485, 'n_contacts': 10, 'rand_seed': 497889}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:45,425] Trial 56 finished with value: 14.70718110473058 and parameters: {'beta': 0.03006066450895288, 'init_prev': 0.039842042699601864, 'n_contacts': 8, 'rand_seed': 511101}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:45,595] Trial 57 finished with value: 12.428082244875213 and parameters: {'beta': 0.03315215231180986, 'init_prev': 0.03903154850114172, 'n_contacts': 8, 'rand_seed': 410367}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:45,886] Trial 58 finished with value: 70.2541425080824 and parameters: {'beta': 0.050601706007675606, 'init_prev': 0.03865911899671829, 'n_contacts': 8, 'rand_seed': 13317}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:46,059] Trial 59 finished with value: 88.64436585910312 and parameters: {'beta': 0.05099288722491598, 'init_prev': 0.02215276923878886, 'n_contacts': 9, 'rand_seed': 840887}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:46,343] Trial 60 finished with value: 149.2137912637006 and parameters: {'beta': 0.06668751389516342, 'init_prev': 0.02249457115795436, 'n_contacts': 9, 'rand_seed': 707702}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:46,517] Trial 61 finished with value: 130.13098430591685 and parameters: {'beta': 0.07263594663549368, 'init_prev': 0.04552120288178532, 'n_contacts': 8, 'rand_seed': 706348}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:46,814] Trial 62 finished with value: 11.785187937758337 and parameters: {'beta': 0.045097070389714354, 'init_prev': 0.02544003156636151, 'n_contacts': 8, 'rand_seed': 752386}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:46,984] Trial 63 finished with value: 23.42412474900334 and parameters: {'beta': 0.045982542467931736, 'init_prev': 0.025794997029036298, 'n_contacts': 8, 'rand_seed': 743208}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:47,225] Trial 64 finished with value: 160.8667078115693 and parameters: {'beta': 0.026657528775527028, 'init_prev': 0.019027965268567287, 'n_contacts': 3, 'rand_seed': 831388}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:47,454] Trial 65 finished with value: 12.8857041746312 and parameters: {'beta': 0.03726667231045534, 'init_prev': 0.017936070318967035, 'n_contacts': 7, 'rand_seed': 832747}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:47,696] Trial 66 finished with value: 35.94419858708238 and parameters: {'beta': 0.05824334606824809, 'init_prev': 0.027171632692133185, 'n_contacts': 7, 'rand_seed': 931135}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:47,912] Trial 67 finished with value: 117.84586707348512 and parameters: {'beta': 0.05963250053510026, 'init_prev': 0.029519614238233783, 'n_contacts': 8, 'rand_seed': 928914}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:48,169] Trial 68 finished with value: 14.431575045206728 and parameters: {'beta': 0.03687704067999107, 'init_prev': 0.029437320304218445, 'n_contacts': 8, 'rand_seed': 362955}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:48,374] Trial 69 finished with value: 180.8122363153268 and parameters: {'beta': 0.08167853709639379, 'init_prev': 0.023361119084928078, 'n_contacts': 9, 'rand_seed': 659514}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:48,626] Trial 70 finished with value: 183.17504326829726 and parameters: {'beta': 0.0816796072798684, 'init_prev': 0.02363980118178439, 'n_contacts': 9, 'rand_seed': 672952}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:48,837] Trial 71 finished with value: 38.621439661758586 and parameters: {'beta': 0.03181288336009084, 'init_prev': 0.027762746953798222, 'n_contacts': 6, 'rand_seed': 140551}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:49,097] Trial 72 finished with value: 19.582861972063938 and parameters: {'beta': 0.04582680149637753, 'init_prev': 0.024683253084802577, 'n_contacts': 8, 'rand_seed': 764873}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:49,305] Trial 73 finished with value: 34.592300084340195 and parameters: {'beta': 0.04623786621650925, 'init_prev': 0.02574753476967111, 'n_contacts': 8, 'rand_seed': 751798}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:49,574] Trial 74 finished with value: 12.876469258095995 and parameters: {'beta': 0.04237081670795706, 'init_prev': 0.025540861289070883, 'n_contacts': 8, 'rand_seed': 745988}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:49,776] Trial 75 finished with value: 8.037015867503897 and parameters: {'beta': 0.03953544217951307, 'init_prev': 0.019526981367653504, 'n_contacts': 8, 'rand_seed': 604277}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:50,037] Trial 76 finished with value: 43.16982910792444 and parameters: {'beta': 0.053933500979434064, 'init_prev': 0.03726765446655622, 'n_contacts': 7, 'rand_seed': 600177}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:50,237] Trial 77 finished with value: 20.6378717562809 and parameters: {'beta': 0.036816905073842277, 'init_prev': 0.016906259393502005, 'n_contacts': 7, 'rand_seed': 598691}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:50,521] Trial 78 finished with value: 12.700472449198742 and parameters: {'beta': 0.03755996767046929, 'init_prev': 0.019504840797945025, 'n_contacts': 7, 'rand_seed': 555452}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:50,715] Trial 79 finished with value: 10.708327287897873 and parameters: {'beta': 0.02851284696757123, 'init_prev': 0.020322658292821677, 'n_contacts': 9, 'rand_seed': 542947}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:51,003] Trial 80 finished with value: 18.92029808088114 and parameters: {'beta': 0.02828372010283174, 'init_prev': 0.013033273873511338, 'n_contacts': 9, 'rand_seed': 475739}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:51,200] Trial 81 finished with value: 14.517642152021608 and parameters: {'beta': 0.02709404956676075, 'init_prev': 0.04096681538734827, 'n_contacts': 9, 'rand_seed': 484486}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:51,476] Trial 82 finished with value: 28.363615191789677 and parameters: {'beta': 0.02433373446459644, 'init_prev': 0.02029156577822774, 'n_contacts': 9, 'rand_seed': 541633}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:51,690] Trial 83 finished with value: 9.849620965419263 and parameters: {'beta': 0.02967405113528104, 'init_prev': 0.016383179923870554, 'n_contacts': 9, 'rand_seed': 524794}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:51,964] Trial 84 finished with value: 11.259590634356528 and parameters: {'beta': 0.03101937624585115, 'init_prev': 0.02208074501429904, 'n_contacts': 10, 'rand_seed': 524296}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:52,173] Trial 85 finished with value: 47.695304303732655 and parameters: {'beta': 0.020403078659579785, 'init_prev': 0.016054749002556958, 'n_contacts': 10, 'rand_seed': 438461}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:52,449] Trial 86 finished with value: 19.24882704632796 and parameters: {'beta': 0.03416767682644724, 'init_prev': 0.016744012764134696, 'n_contacts': 10, 'rand_seed': 636859}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:52,653] Trial 87 finished with value: 19.033305034871546 and parameters: {'beta': 0.029062267620282346, 'init_prev': 0.01160358972812348, 'n_contacts': 9, 'rand_seed': 639341}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:52,933] Trial 88 finished with value: 13.812667629594898 and parameters: {'beta': 0.029361509368987307, 'init_prev': 0.011877955208478343, 'n_contacts': 9, 'rand_seed': 527809}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:53,128] Trial 89 finished with value: 97.08372800263965 and parameters: {'beta': 0.01836742118997984, 'init_prev': 0.014724465296360444, 'n_contacts': 9, 'rand_seed': 579205}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:53,388] Trial 90 finished with value: 123.24410733580658 and parameters: {'beta': 0.02226303912606713, 'init_prev': 0.013917689183450932, 'n_contacts': 6, 'rand_seed': 437339}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:53,599] Trial 91 finished with value: 81.84071796794592 and parameters: {'beta': 0.02222362073792625, 'init_prev': 0.015813544563027755, 'n_contacts': 8, 'rand_seed': 435624}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:53,887] Trial 92 finished with value: 8.272659408811478 and parameters: {'beta': 0.03165329338404759, 'init_prev': 0.021824156594044828, 'n_contacts': 10, 'rand_seed': 506750}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:54,086] Trial 93 finished with value: 15.98823808791417 and parameters: {'beta': 0.02536352025771004, 'init_prev': 0.019542823455539603, 'n_contacts': 10, 'rand_seed': 514565}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:54,379] Trial 94 finished with value: 12.277148410691325 and parameters: {'beta': 0.026011272636754675, 'init_prev': 0.018375608916912704, 'n_contacts': 10, 'rand_seed': 502000}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:54,571] Trial 95 finished with value: 40.39785813378671 and parameters: {'beta': 0.04088860187058547, 'init_prev': 0.01817611955436269, 'n_contacts': 10, 'rand_seed': 492864}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:54,858] Trial 96 finished with value: 42.317263825134205 and parameters: {'beta': 0.03979563000970366, 'init_prev': 0.03442343464240234, 'n_contacts': 9, 'rand_seed': 555250}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:55,057] Trial 97 finished with value: 11.861324489804929 and parameters: {'beta': 0.032607918586922656, 'init_prev': 0.021299117268899607, 'n_contacts': 9, 'rand_seed': 609400}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:55,340] Trial 98 finished with value: 12.242714793725781 and parameters: {'beta': 0.03455699447663427, 'init_prev': 0.032707969395604476, 'n_contacts': 8, 'rand_seed': 597573}. Best is trial 32 with value: 7.544700950781362.
[I 2024-11-21 04:20:55,473] Trial 99 finished with value: 7.6694276848072604 and parameters: {'beta': 0.03432569464640182, 'init_prev': 0.03761708876991067, 'n_contacts': 8, 'rand_seed': 584671}. Best is trial 32 with value: 7.544700950781362.
Making results structure...
Processed 100 trials; 0 failed
Best pars: {'beta': 0.03798904967029662, 'init_prev': 0.0275687855655205, 'n_contacts': 8, 'rand_seed': 647595}
Removed existing calibration file starsim_calibration.db
Let’s look at the best parameters that were found. Note that the rand_seed was selected at random, but the other parameters are meaningful.
[7]:
calib.best_pars
[7]:
{'beta': 0.03798904967029662,
'init_prev': 0.0275687855655205,
'n_contacts': 8,
'rand_seed': 647595}
Once the calibration is complete, we can compare the guess values to the best values found by calling check_fit.
[8]:
# Confirm
sc.printcyan('\nConfirming fit...')
calib.check_fit(n_runs=5)
Confirming fit...
Checking fit...
Elapsed time: 1.47 s
Elapsed time: 1.64 s
Fit with original pars: [102.88081432 103.17668444 101.76293553 101.73024849 101.42137967]
Fit with best-fit pars: [16.12044405 7.76731953 11.57553006 7.98078489 13.98970462]
✓ Calibration improved fit
[8]:
(array([102.88081432, 103.17668444, 101.76293553, 101.73024849,
101.42137967]),
array([16.12044405, 7.76731953, 11.57553006, 7.98078489, 13.98970462]))
Finally, we can view some plots of the results. Blue is before calibration using the guess values whereas orange is after.
[9]:
calib.plot_sims()
[9]:
(<Figure size 933.333x700 with 12 Axes>,
<Figure size 933.333x700 with 12 Axes>)
[10]:
calib.plot_trend()
[10]:
