T7 - Calibration

An interactive version of this notebook is available on Google Colab or Binder.

Disease models typically require contextualization to a relevant setting of interest prior to addressing “what-if” scenario questions. The process of tuning model input parameters so that model outputs match observed data is known as calibration. There are many approaches to model calibration, ranging from manual tuning to fully Bayesian methods.

For many applications, we have found that an optimization-based approach is sufficient. Such methods avoid the tedious process of manual tuning and are less computationally expensive than fully Bayesian methods. One such optimization-based approach is the Optuna library, which is a Bayesian hyperparameter optimization framework. Optuna is designed for tuning hyperparameters of machine learning models, but it can also be used to calibrate disease models.

Calibration libraries often treat the disease model as a black box, where the input parameters are the “hyperparameters” to be tuned. The calibration process is often iterative and requires a combination of expert knowledge and computational tools. The optimization algorithm iteratively chooses new parameter values to evaluate, and the model is run with these values to generate outputs. The outputs are compared to observed data, and a loss function is calculated to quantify the difference between the model outputs and the observed data. The optimization algorithm then uses this loss function to update its search strategy and choose new parameter values to evaluate. This process continues until the algorithm converges to a set of parameter values that minimize the loss function.

While many optimization algorithms are available, Starsim has a built-in interface to the Optuna library, which we will demonstrate in this tutorial. We will use a simple Susceptible-Infected-Recovered (SIR) model as an example. We will tune three input parameters, the infectivity parameter, beta, the initial prevalence parameter, init_prev, and the Poisson-distributed degree distribution parameter, n_contacts. We will calibrate the model using a beta-binomial likelihood function so as to match prevalence at three distinct time points.

We begin with a few imports and default settings:

#%% Imports and settings
import sciris as sc
import starsim as ss
import numpy as np
import pandas as pd
import matplotlib.dates as mdates

n_agents = 2e3
debug = False # If true, will run in serial

The calibration class will require a base Sim object. This sim will later be modified according to parameters selected by the optimization engine. The following function creates the base Sim object.

def make_sim():
    sir = ss.SIR(
        beta = ss.beta(0.075),
        init_prev = ss.bernoulli(0.02),
    )
    random = ss.RandomNet(n_contacts=ss.poisson(4))

    sim = ss.Sim(
        n_agents = n_agents,
        start = sc.date('2020-01-01'),
        stop = sc.date('2020-02-12'),
        dt = 1,
        unit = 'day',
        diseases = sir,
        networks = random,
        verbose = 0,
    )

    return sim

Now let’s define the calibration parameters. These are the inputs that Optuna will be able to modify. Here, we define three such parameters, beta, init_prev, and n_contacts.

Each parameter entry should have range defined by low and high as well as a guess values. The guess value is not used by Optuna, rather only for a check after calibration completes to see if the new parameters are better than the guess values.

You’ll notice there are a few other parameters that can be specified. For example, the data type of the parameter appears in suggest_type. Possible values are listed in the Optuna documentation, and include suggest_float for float values and suggest_int for integer types.

To make things easier for the search algorithm, it’s helpful to indicate how outputs are expected to change with inputs. For example, increasing beta from 0.01 to 0.02 should double disease transmission, but increasing from 0.11 to 0.12 will have a small effect. Thus, we indicate that this parameter should be calibrated with log=True.

# Define the calibration parameters
calib_pars = dict(
    beta = dict(low=0.01, high=0.30, guess=0.15, suggest_type='suggest_float', log=True), # Note the log scale
    init_prev = dict(low=0.01, high=0.05, guess=0.15), # Default type is suggest_float, no need to re-specify
    n_contacts = dict(low=2, high=10, guess=3, suggest_type='suggest_int'), # Suggest int just for this demo
)

The optimization engine iteratively chooses input parameters to simulate. Those parameters are passed into the following build_sim function as a dictionary of calib_pars along with the base sim and any other key word arguments. The calib_pars will be as above, but importantly will have an additional key named value containing the value selected by Optuna.

When modifying a sim, it is important to realize that the simulation has not been initialized yet. Nonetheless, the configuration is available for modification at sim.pars, as demonstrated in the function below for the SIR example.

def build_sim(sim, calib_pars, n_reps=1, **kwargs):
    """
    Modify the base simulation by applying calib_pars. The result can be a
    single simulation or multiple simulations if n_reps>1. Note that here we are
    simply building the simulation by modifying the base sim. Running the sims
    and extracting results will be done by the calibration function.
    """

    sir = sim.pars.diseases # There is only one disease in this simulation and it is a SIR
    net = sim.pars.networks # There is only one network in this simulation and it is a RandomNet

    for k, pars in calib_pars.items(): # Loop over the calibration parameters
        if k == 'rand_seed':
            sim.pars.rand_seed = v
            continue

        # Each item in calib_pars is a dictionary with keys like 'low', 'high',
        # 'guess', 'suggest_type', and importantly 'value'. The 'value' key is
        # the one we want to use as that's the one selected by the algorithm
        v = pars['value']
        if k == 'beta':
            sir.pars.beta = ss.beta(v)
        elif k == 'init_prev':
            sir.pars.init_prev = ss.bernoulli(v)
        elif k == 'n_contacts':
            net.pars.n_contacts = ss.poisson(v)
        else:
            raise NotImplementedError(f'Parameter {k} not recognized')

    # If just one simulation per parameter set, return the single simulation
    if n_reps == 1:
        return sim

    # But if you'd like to run multiple simulations with the same parameters, we return a MultiSim instead
    # Note that each simulation will have a different random seed, you can set specific seeds if you like
    # Also note that parallel=False and debug=True are important to avoid issues with parallelism in the calibration
    # Advanced: If running multiple reps, you can choose if/how they are combined using the "combine_reps" argument to each CalibComponent, introduced below.
    ms = ss.MultiSim(sim, iterpars=dict(rand_seed=np.random.randint(0, 1e6, n_reps)), initialize=True, debug=True, parallel=False)
    return ms

The Starsim framework has been integrated with the Optuna hyperparameter optimization algorithm to facilitate calibration through the Calibration class. Recall that an optimization-based approach to calibration minimizes a function of the input parameters. This function is key to achieving an acceptable calibration.

There are two ways to describe the goodness-of-fit function for the Calibration. The first method is to directly provide a function that the algorithm will call. The eval_fn will be passed each completed sim after running, and is expected to return a float representing the mismatch (lower is better as the optimization algorithm is configured to minimize). Data can be passed into the eval_fn via eval_kwargs.

As an alternative to directly specifying the evaluation function, you can use CalibComponents. Each component includes real data, for example from a survey, that is compared against simulation data from the model. Several components and be used at the same time, for example one for disease prevalence and another for treatment coverage. Each component computes a likelihood of the data given the input parameters, as assessed via simulation. Components are combined assuming independence.

The base class for a component is called CalibComponent, which you can use to define your own likelihood. However, we have provided components for several key likelihood functions including BetaBinomial, Binomial, DirichletMultinomial, GammaPoisson, and Normal. The Normal component is most like a traditional squared error. Each component takes in a name and a weight, which is used when combining log likelihoods.

Importantly, each component takes in the calibration target, the real data that was observed, in an argument called expected. This argument should be a Pandas Dataframe with one row per time point and columns that will depend on the specific component type. For example, the Binomial component requires columns of n (trials) and x (successes).

The components also handle extracting data from each simulation using the extract_fn argument. The value of this argument should be a function that takes in a simulation and returns a Pandas DataFrame. The specifics of the columns will depend a bit on the type of component (e.g. BetaBinomial is different from Normal), but often looks like a simulated version of expected. We will see examples below.

We’ll also see how to use the conform argument, the purpose of which is to temporally align the simulation output to the real data. This argument works along with the extract_fn to produce the final simulation outputs that are used in the likelihood function. The conformer is a function that takes in the expected data you provided and the actual simulation result the comes out of the extract_fn. The conformers we have built in are as follows:

• step_containing: Conform by simply choosing the simulated timestep that contains the time indicated in the real data (expected)

• prevalent: Interpolate the simulated timepoints to estimate the values that would have occurred at each real timepoint

• incident: While the two methods above capture the state of the model at a particular point in time (stocks), this component allows you to capture the behavior of the model over time (flows). Instead of just giving one time value, t, you’ll provide a second time value as well called t1. This conformer will add up events occurring between the two time points.

Let’s make a Binomial component, as might be used to calibrate disease prevalence.

prevalence = ss.Normal(
    name = 'Disease prevalence',
    conform = 'prevalent',

    expected = pd.DataFrame({
        'x': [0.13, 0.16, 0.06],    # Prevalence of infection
    }, index=pd.Index([ss.date(d) for d in ['2020-01-12', '2020-01-25', '2020-02-02']], name='t')), # On these dates

    extract_fn = lambda sim: pd.DataFrame({
        'x': sim.results.sir.prevalence,
    }, index=pd.Index(sim.results.timevec, name='t')),

    # You can specify the variance as well, but it's optional (max likelihood estimates will be used if not provided)
    # This could be a single float or an array with the same shape as the expected values
    sigma2 = 0.05, # e.g. (num_replicates/sigma2_model + 1/sigma2_data)^-1
    #sigma2 = np.array([0.05, 0.25, 0.01])
)

Finally, we can bring all the pieces together. We make a single base simulation and create an instance of a Starsim Calibration object. This object requires a few arguments, like the calib_pars and sim. We also pass in the function that modifies the base sim, here our build_sim function. No additional build_kw are required in this example.

We also pass in a list of components. Instead of using this “component-based” system, a user could simply provide an eval_fn, which takes in a completed sim an any eval_kwargs and returns a “mismatch” score to be minimized.

We can also specify the total number of trials to run, the number of parallel works, and a few other parameters.

sc.heading('Beginning calibration')

# Make the sim and data
sim = make_sim()

# Make the calibration
calib = ss.Calibration(
    calib_pars = calib_pars,
    sim = sim,
    build_fn = build_sim,
    build_kw = dict(n_reps=3), # Run 3 replicates for each parameter set
    reseed = True, # If true, a different random seed will be provided to each configuration
    components = [prevalence],
    total_trials = 100,
    n_workers = None, # None indicates to use all available CPUs
    die = True,
    debug = debug, # Run in serial if True
)

# Perform the calibration
sc.printcyan('\nPeforming calibration...')
calib.calibrate();

—————————————————————
Beginning calibration
—————————————————————


Peforming calibration...
Removed existing calibration file starsim_calibration.db
sqlite:///starsim_calibration.db

[I 2025-02-14 05:57:42,231] A new study created in RDB with name: starsim_calibration

Elapsed time: 0.786 s

[I 2025-02-14 05:57:43,665] Trial 0 finished with value: -0.4371241893751869 and parameters: {'beta': 0.012084575581731411, 'init_prev': 0.023129508821673014, 'n_contacts': 9, 'rand_seed': 470890}. Best is trial 0 with value: -0.4371241893751869.

Elapsed time: 0.783 s

[I 2025-02-14 05:57:43,746] Trial 1 finished with value: -0.4687596278120426 and parameters: {'beta': 0.2086494313387353, 'init_prev': 0.02021608236593667, 'n_contacts': 10, 'rand_seed': 63123}. Best is trial 1 with value: -0.4687596278120426.

Elapsed time: 0.685 s

[I 2025-02-14 05:57:44,480] Trial 2 finished with value: -0.4547220425372189 and parameters: {'beta': 0.043674490976145876, 'init_prev': 0.02214418001887687, 'n_contacts': 3, 'rand_seed': 34630}. Best is trial 1 with value: -0.4687596278120426.

Elapsed time: 0.720 s

[I 2025-02-14 05:57:44,597] Trial 3 finished with value: -0.48055520183491535 and parameters: {'beta': 0.19848660230173087, 'init_prev': 0.02400821478940289, 'n_contacts': 9, 'rand_seed': 553598}. Best is trial 3 with value: -0.48055520183491535.

Elapsed time: 0.642 s
Elapsed time: 0.764 s

[I 2025-02-14 05:57:45,369] Trial 5 finished with value: -0.425836151178834 and parameters: {'beta': 0.010049453232426231, 'init_prev': 0.010677991040314968, 'n_contacts': 3, 'rand_seed': 30392}. Best is trial 3 with value: -0.48055520183491535.
[I 2025-02-14 05:57:45,384] Trial 4 finished with value: -0.516521236253933 and parameters: {'beta': 0.02723757741344654, 'init_prev': 0.013934674723208033, 'n_contacts': 8, 'rand_seed': 303287}. Best is trial 4 with value: -0.516521236253933.

Elapsed time: 0.743 s

[I 2025-02-14 05:57:46,247] Trial 6 finished with value: -0.5251629173146466 and parameters: {'beta': 0.04029674359691771, 'init_prev': 0.04100111203471901, 'n_contacts': 5, 'rand_seed': 231023}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.798 s

[I 2025-02-14 05:57:46,348] Trial 7 finished with value: -0.4836822588644354 and parameters: {'beta': 0.0329045664318984, 'init_prev': 0.038676819119359294, 'n_contacts': 10, 'rand_seed': 831542}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.899 s

[I 2025-02-14 05:57:47,282] Trial 8 finished with value: -0.43799621468343386 and parameters: {'beta': 0.024050860369204508, 'init_prev': 0.011581148843609368, 'n_contacts': 6, 'rand_seed': 195937}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.723 s

[I 2025-02-14 05:57:47,405] Trial 9 finished with value: 0.3630192155995597 and parameters: {'beta': 0.09220905096930561, 'init_prev': 0.02043923937676122, 'n_contacts': 7, 'rand_seed': 954227}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.720 s

[I 2025-02-14 05:57:48,140] Trial 10 finished with value: -0.3091627989447342 and parameters: {'beta': 0.09981265793339471, 'init_prev': 0.04928924305785637, 'n_contacts': 8, 'rand_seed': 936528}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.720 s

[I 2025-02-14 05:57:48,282] Trial 11 finished with value: -0.17582963945761376 and parameters: {'beta': 0.08527514683566394, 'init_prev': 0.04963768163574096, 'n_contacts': 5, 'rand_seed': 556925}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.688 s

[I 2025-02-14 05:57:48,986] Trial 12 finished with value: -0.44398608925651334 and parameters: {'beta': 0.02078446756122935, 'init_prev': 0.036312712135873955, 'n_contacts': 5, 'rand_seed': 332822}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.690 s

[I 2025-02-14 05:57:49,125] Trial 13 finished with value: -0.43840764988330894 and parameters: {'beta': 0.02039047070694797, 'init_prev': 0.03772152935624125, 'n_contacts': 5, 'rand_seed': 310514}. Best is trial 6 with value: -0.5251629173146466.

Elapsed time: 0.744 s

[I 2025-02-14 05:57:49,890] Trial 14 finished with value: -0.5633738310800133 and parameters: {'beta': 0.053496489306436454, 'init_prev': 0.03869483589415642, 'n_contacts': 5, 'rand_seed': 296600}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.745 s

[I 2025-02-14 05:57:50,027] Trial 15 finished with value: -0.391374114863332 and parameters: {'beta': 0.05373969149149277, 'init_prev': 0.0302378397675459, 'n_contacts': 7, 'rand_seed': 237054}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.644 s

[I 2025-02-14 05:57:50,691] Trial 16 finished with value: -0.44621410368188386 and parameters: {'beta': 0.05452320089206155, 'init_prev': 0.04284886689495206, 'n_contacts': 2, 'rand_seed': 186823}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.701 s

[I 2025-02-14 05:57:50,884] Trial 17 finished with value: -0.5452665938202881 and parameters: {'beta': 0.1435982292645229, 'init_prev': 0.04404603895452439, 'n_contacts': 2, 'rand_seed': 426144}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.698 s

[I 2025-02-14 05:57:51,548] Trial 18 finished with value: 0.030503182299610306 and parameters: {'beta': 0.13457828115415044, 'init_prev': 0.04328400526090588, 'n_contacts': 4, 'rand_seed': 444349}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.675 s

[I 2025-02-14 05:57:51,715] Trial 19 finished with value: 0.17155047564482262 and parameters: {'beta': 0.2996014821704118, 'init_prev': 0.03026909750768729, 'n_contacts': 2, 'rand_seed': 435191}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.714 s

[I 2025-02-14 05:57:52,422] Trial 20 finished with value: -0.5595560153850507 and parameters: {'beta': 0.14664605152227703, 'init_prev': 0.03218767326719892, 'n_contacts': 2, 'rand_seed': 705057}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.721 s

[I 2025-02-14 05:57:52,599] Trial 21 finished with value: -0.5484468393945644 and parameters: {'beta': 0.0760762204507974, 'init_prev': 0.03328284884623598, 'n_contacts': 3, 'rand_seed': 672101}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.699 s

[I 2025-02-14 05:57:53,280] Trial 22 finished with value: -0.40891472371501764 and parameters: {'beta': 0.12357646229522967, 'init_prev': 0.033549363957611655, 'n_contacts': 3, 'rand_seed': 697142}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.711 s

[I 2025-02-14 05:57:53,469] Trial 23 finished with value: -0.5449439024793478 and parameters: {'beta': 0.07866767283121644, 'init_prev': 0.03307114584359148, 'n_contacts': 3, 'rand_seed': 702139}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.723 s

[I 2025-02-14 05:57:54,166] Trial 24 finished with value: -0.5372968499225887 and parameters: {'beta': 0.07613903285102543, 'init_prev': 0.03266450554187257, 'n_contacts': 4, 'rand_seed': 685115}. Best is trial 14 with value: -0.5633738310800133.

Elapsed time: 0.729 s

[I 2025-02-14 05:57:54,359] Trial 25 finished with value: -0.5685757827604765 and parameters: {'beta': 0.06671102065885659, 'init_prev': 0.02669566760210894, 'n_contacts': 4, 'rand_seed': 672302}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.727 s

[I 2025-02-14 05:57:55,056] Trial 26 finished with value: -0.5664373406522808 and parameters: {'beta': 0.0659798760402135, 'init_prev': 0.026629335814873673, 'n_contacts': 4, 'rand_seed': 828490}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.695 s

[I 2025-02-14 05:57:55,213] Trial 27 finished with value: -0.3619709831179432 and parameters: {'beta': 0.18452614375088577, 'init_prev': 0.02623328924621884, 'n_contacts': 6, 'rand_seed': 799726}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.725 s

[I 2025-02-14 05:57:55,941] Trial 28 finished with value: -0.562634399370051 and parameters: {'beta': 0.06513762896894337, 'init_prev': 0.026163226955099084, 'n_contacts': 4, 'rand_seed': 815895}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.734 s

[I 2025-02-14 05:57:56,105] Trial 29 finished with value: -0.5509828947179318 and parameters: {'beta': 0.061300578979107366, 'init_prev': 0.016638570133453705, 'n_contacts': 4, 'rand_seed': 843319}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.651 s

[I 2025-02-14 05:57:56,749] Trial 30 finished with value: -0.42842962458109013 and parameters: {'beta': 0.015079585460150688, 'init_prev': 0.02655262216767426, 'n_contacts': 4, 'rand_seed': 538093}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.692 s

[I 2025-02-14 05:57:56,958] Trial 31 finished with value: -0.4357121549919609 and parameters: {'beta': 0.014741048991165273, 'init_prev': 0.027104244120917706, 'n_contacts': 6, 'rand_seed': 589179}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.714 s

[I 2025-02-14 05:57:57,623] Trial 32 finished with value: -0.4789090341369468 and parameters: {'beta': 0.043661678130082225, 'init_prev': 0.026712183265150853, 'n_contacts': 4, 'rand_seed': 901743}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.725 s

[I 2025-02-14 05:57:57,849] Trial 33 finished with value: -0.5645304890806406 and parameters: {'beta': 0.06224418884316947, 'init_prev': 0.02424757668166499, 'n_contacts': 4, 'rand_seed': 775224}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.732 s

[I 2025-02-14 05:57:58,515] Trial 34 finished with value: -0.5683994552554796 and parameters: {'beta': 0.061728461175429655, 'init_prev': 0.01816583592107047, 'n_contacts': 5, 'rand_seed': 791426}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.735 s

[I 2025-02-14 05:57:58,748] Trial 35 finished with value: -0.49938299277521875 and parameters: {'beta': 0.03693675616751872, 'init_prev': 0.01812577301017424, 'n_contacts': 5, 'rand_seed': 753998}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.729 s

[I 2025-02-14 05:57:59,404] Trial 36 finished with value: -0.47249041691352317 and parameters: {'beta': 0.035603565953620166, 'init_prev': 0.016420582557084024, 'n_contacts': 5, 'rand_seed': 774452}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.724 s

[I 2025-02-14 05:57:59,636] Trial 37 finished with value: 0.2328263391338884 and parameters: {'beta': 0.10714308219787574, 'init_prev': 0.023112429566213996, 'n_contacts': 7, 'rand_seed': 628336}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.728 s

[I 2025-02-14 05:58:00,294] Trial 38 finished with value: 0.30519610362912025 and parameters: {'beta': 0.10161052198204333, 'init_prev': 0.01814009940811441, 'n_contacts': 6, 'rand_seed': 614737}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.733 s

[I 2025-02-14 05:58:00,532] Trial 39 finished with value: -0.384374346074231 and parameters: {'beta': 0.06598187403184717, 'init_prev': 0.021342312871507244, 'n_contacts': 6, 'rand_seed': 871248}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.646 s

[I 2025-02-14 05:58:01,105] Trial 40 finished with value: -0.43101199310040617 and parameters: {'beta': 0.029457104650902392, 'init_prev': 0.021435681412981765, 'n_contacts': 3, 'rand_seed': 988459}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.673 s

[I 2025-02-14 05:58:01,375] Trial 41 finished with value: -0.44446751574954274 and parameters: {'beta': 0.04810373211858719, 'init_prev': 0.024438856773782845, 'n_contacts': 3, 'rand_seed': 757248}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.729 s

[I 2025-02-14 05:58:02,001] Trial 42 finished with value: -0.5206518575646606 and parameters: {'beta': 0.04970297901219555, 'init_prev': 0.02406211590468409, 'n_contacts': 4, 'rand_seed': 760669}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.737 s

[I 2025-02-14 05:58:02,281] Trial 43 finished with value: -0.5574248323274765 and parameters: {'beta': 0.04837690447834074, 'init_prev': 0.028654745498395317, 'n_contacts': 5, 'rand_seed': 875092}. Best is trial 25 with value: -0.5685757827604765.

Elapsed time: 0.742 s

[I 2025-02-14 05:58:02,921] Trial 44 finished with value: -0.570660996347727 and parameters: {'beta': 0.06289516721459826, 'init_prev': 0.01358605461280269, 'n_contacts': 5, 'rand_seed': 99089}. Best is trial 44 with value: -0.570660996347727.

Elapsed time: 0.739 s

[I 2025-02-14 05:58:03,188] Trial 45 finished with value: -0.5723973364116521 and parameters: {'beta': 0.06427026468969946, 'init_prev': 0.013731546394239607, 'n_contacts': 5, 'rand_seed': 370845}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.723 s

[I 2025-02-14 05:58:03,808] Trial 46 finished with value: -0.561545733198184 and parameters: {'beta': 0.0651615341442023, 'init_prev': 0.012754501978829652, 'n_contacts': 4, 'rand_seed': 126327}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.734 s

[I 2025-02-14 05:58:04,090] Trial 47 finished with value: -0.5525384200649863 and parameters: {'beta': 0.06993869400541895, 'init_prev': 0.013313782021853801, 'n_contacts': 5, 'rand_seed': 106693}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.748 s

[I 2025-02-14 05:58:04,721] Trial 48 finished with value: -0.5376868284766609 and parameters: {'beta': 0.03893585923869891, 'init_prev': 0.01423163341598733, 'n_contacts': 6, 'rand_seed': 77507}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.728 s

[I 2025-02-14 05:58:04,989] Trial 49 finished with value: 0.4121564971728391 and parameters: {'beta': 0.09167664184388422, 'init_prev': 0.014863858323701804, 'n_contacts': 7, 'rand_seed': 15049}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.727 s

[I 2025-02-14 05:58:05,615] Trial 50 finished with value: 0.3956981580655144 and parameters: {'beta': 0.08875460930804921, 'init_prev': 0.01908772899525451, 'n_contacts': 7, 'rand_seed': 375499}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.725 s

[I 2025-02-14 05:58:05,883] Trial 51 finished with value: -0.33169541781034034 and parameters: {'beta': 0.0871041238690469, 'init_prev': 0.019031955764045747, 'n_contacts': 5, 'rand_seed': 384567}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.733 s

[I 2025-02-14 05:58:06,516] Trial 52 finished with value: -0.5519985917952567 and parameters: {'beta': 0.05729875582065992, 'init_prev': 0.010348194572739226, 'n_contacts': 5, 'rand_seed': 493345}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.726 s

[I 2025-02-14 05:58:06,776] Trial 53 finished with value: -0.54164706842834 and parameters: {'beta': 0.058980405265598564, 'init_prev': 0.01155467416549149, 'n_contacts': 4, 'rand_seed': 490680}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.714 s

[I 2025-02-14 05:58:07,398] Trial 54 finished with value: -0.4850109159571285 and parameters: {'beta': 0.04264278894026143, 'init_prev': 0.028815909604356878, 'n_contacts': 4, 'rand_seed': 918597}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.730 s

[I 2025-02-14 05:58:07,672] Trial 55 finished with value: -0.11026138191644848 and parameters: {'beta': 0.11355423170502624, 'init_prev': 0.01649922891150989, 'n_contacts': 9, 'rand_seed': 634243}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.735 s

[I 2025-02-14 05:58:08,302] Trial 56 finished with value: -0.5569271981136712 and parameters: {'beta': 0.0724499766923919, 'init_prev': 0.015562616216025968, 'n_contacts': 5, 'rand_seed': 731207}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.733 s

[I 2025-02-14 05:58:08,583] Trial 57 finished with value: -0.5021882185564572 and parameters: {'beta': 0.07282084053920711, 'init_prev': 0.022789205646569825, 'n_contacts': 5, 'rand_seed': 736181}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.744 s

[I 2025-02-14 05:58:09,213] Trial 58 finished with value: -0.49676735638142616 and parameters: {'beta': 0.0320129694977849, 'init_prev': 0.022428625405773644, 'n_contacts': 6, 'rand_seed': 224684}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.650 s

[I 2025-02-14 05:58:09,403] Trial 59 finished with value: -0.4337703782474152 and parameters: {'beta': 0.03188326089771193, 'init_prev': 0.025134634253929776, 'n_contacts': 3, 'rand_seed': 169640}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.697 s

[I 2025-02-14 05:58:10,078] Trial 60 finished with value: -0.45710742970613893 and parameters: {'beta': 0.05044293991137874, 'init_prev': 0.019892665590218638, 'n_contacts': 3, 'rand_seed': 796746}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.919 s

[I 2025-02-14 05:58:10,493] Trial 61 finished with value: -0.47612303854945254 and parameters: {'beta': 0.05384188961610545, 'init_prev': 0.01222406386228957, 'n_contacts': 4, 'rand_seed': 801483}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.949 s

[I 2025-02-14 05:58:11,191] Trial 62 finished with value: -0.5315186446321741 and parameters: {'beta': 0.0558914980700272, 'init_prev': 0.0407787847241878, 'n_contacts': 5, 'rand_seed': 284919}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.725 s

[I 2025-02-14 05:58:11,391] Trial 63 finished with value: -0.246977626280437 and parameters: {'beta': 0.08172887560617058, 'init_prev': 0.04785789188054382, 'n_contacts': 5, 'rand_seed': 302388}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.727 s

[I 2025-02-14 05:58:12,091] Trial 64 finished with value: -0.25465404342765025 and parameters: {'beta': 0.08049828046187799, 'init_prev': 0.04564381481486385, 'n_contacts': 5, 'rand_seed': 328645}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.732 s

[I 2025-02-14 05:58:12,296] Trial 65 finished with value: -0.5576624119200918 and parameters: {'beta': 0.062219238277575195, 'init_prev': 0.031055720859528905, 'n_contacts': 4, 'rand_seed': 371276}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.736 s

[I 2025-02-14 05:58:13,003] Trial 66 finished with value: -0.5526367000173075 and parameters: {'beta': 0.06343645055896202, 'init_prev': 0.03525295237662314, 'n_contacts': 4, 'rand_seed': 261167}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.759 s

[I 2025-02-14 05:58:13,231] Trial 67 finished with value: -0.5590688023743158 and parameters: {'beta': 0.04053174993127911, 'init_prev': 0.035548952926784616, 'n_contacts': 6, 'rand_seed': 848325}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.769 s

[I 2025-02-14 05:58:13,944] Trial 68 finished with value: -0.5704141409257333 and parameters: {'beta': 0.04581793876034334, 'init_prev': 0.028243312840949497, 'n_contacts': 6, 'rand_seed': 522971}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.760 s

[I 2025-02-14 05:58:14,166] Trial 69 finished with value: -0.5503471874059822 and parameters: {'beta': 0.04638639448244023, 'init_prev': 0.03787431956217208, 'n_contacts': 5, 'rand_seed': 533438}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.730 s

[I 2025-02-14 05:58:14,853] Trial 70 finished with value: -0.45647549748533955 and parameters: {'beta': 0.024675369728798264, 'init_prev': 0.028401520940990906, 'n_contacts': 6, 'rand_seed': 554217}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.731 s

[I 2025-02-14 05:58:15,078] Trial 71 finished with value: 0.11157115322485452 and parameters: {'beta': 0.09530907588567408, 'init_prev': 0.028311888335673573, 'n_contacts': 6, 'rand_seed': 659886}. Best is trial 45 with value: -0.5723973364116521.

Elapsed time: 0.753 s

[I 2025-02-14 05:58:15,781] Trial 72 finished with value: -0.5749499034003444 and parameters: {'beta': 0.0549476616412528, 'init_prev': 0.0277422478690141, 'n_contacts': 5, 'rand_seed': 666971}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.747 s

[I 2025-02-14 05:58:16,000] Trial 73 finished with value: -0.5411546001246484 and parameters: {'beta': 0.06896224800827007, 'init_prev': 0.025303096939127873, 'n_contacts': 5, 'rand_seed': 456194}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.744 s

[I 2025-02-14 05:58:16,703] Trial 74 finished with value: -0.5668133902434415 and parameters: {'beta': 0.07093373127548229, 'init_prev': 0.02533030073145338, 'n_contacts': 4, 'rand_seed': 587460}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.731 s

[I 2025-02-14 05:58:16,905] Trial 75 finished with value: -0.5243932528161572 and parameters: {'beta': 0.052564458256236866, 'init_prev': 0.029656227035853987, 'n_contacts': 4, 'rand_seed': 651965}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.724 s

[I 2025-02-14 05:58:17,599] Trial 76 finished with value: -0.48630246509201785 and parameters: {'beta': 0.04637886659562189, 'init_prev': 0.030300246646159157, 'n_contacts': 4, 'rand_seed': 589349}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.726 s

[I 2025-02-14 05:58:17,805] Trial 77 finished with value: -0.4908731045277771 and parameters: {'beta': 0.0434780165718067, 'init_prev': 0.031219441176910376, 'n_contacts': 4, 'rand_seed': 596825}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.752 s

[I 2025-02-14 05:58:18,523] Trial 78 finished with value: 0.10838031749389611 and parameters: {'beta': 0.07522241695340474, 'init_prev': 0.032085355778620483, 'n_contacts': 8, 'rand_seed': 576644}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.758 s

[I 2025-02-14 05:58:18,739] Trial 79 finished with value: -0.07513257911585897 and parameters: {'beta': 0.059363518882224964, 'init_prev': 0.027889721600901696, 'n_contacts': 8, 'rand_seed': 714307}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.751 s

[I 2025-02-14 05:58:19,446] Trial 80 finished with value: -0.4801486448191263 and parameters: {'beta': 0.05658551534838699, 'init_prev': 0.027414163731334124, 'n_contacts': 6, 'rand_seed': 519436}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.752 s

[I 2025-02-14 05:58:19,663] Trial 81 finished with value: -0.5194082408482223 and parameters: {'beta': 0.03516536826303736, 'init_prev': 0.02720562651908051, 'n_contacts': 6, 'rand_seed': 524498}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.728 s

[I 2025-02-14 05:58:20,352] Trial 82 finished with value: -0.5611538070777938 and parameters: {'beta': 0.08200086640882402, 'init_prev': 0.02525262988166564, 'n_contacts': 3, 'rand_seed': 671320}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.714 s

[I 2025-02-14 05:58:20,552] Trial 83 finished with value: -0.5140895848704565 and parameters: {'beta': 0.06670836018130837, 'init_prev': 0.025859199163650894, 'n_contacts': 3, 'rand_seed': 417507}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.739 s

[I 2025-02-14 05:58:21,265] Trial 84 finished with value: -0.5692336124039362 and parameters: {'beta': 0.07041404751091426, 'init_prev': 0.023738838250012283, 'n_contacts': 4, 'rand_seed': 818643}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.789 s

[I 2025-02-14 05:58:21,520] Trial 85 finished with value: 0.28503257534644266 and parameters: {'beta': 0.06092851436071887, 'init_prev': 0.02069388602367989, 'n_contacts': 10, 'rand_seed': 886869}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.759 s

[I 2025-02-14 05:58:22,206] Trial 86 finished with value: -0.5279027095227603 and parameters: {'beta': 0.07165128605521753, 'init_prev': 0.021138187165660093, 'n_contacts': 5, 'rand_seed': 880929}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.720 s

[I 2025-02-14 05:58:22,424] Trial 87 finished with value: 0.21744398220340785 and parameters: {'beta': 0.11303053736351015, 'init_prev': 0.023516793834849257, 'n_contacts': 5, 'rand_seed': 823316}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.724 s

[I 2025-02-14 05:58:23,119] Trial 88 finished with value: -0.14997074371109376 and parameters: {'beta': 0.11897832848444095, 'init_prev': 0.023974277557635065, 'n_contacts': 4, 'rand_seed': 834558}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.741 s

[I 2025-02-14 05:58:23,351] Trial 89 finished with value: -0.5052915632896752 and parameters: {'beta': 0.051423802051791714, 'init_prev': 0.01728662285885125, 'n_contacts': 4, 'rand_seed': 622769}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.760 s

[I 2025-02-14 05:58:24,069] Trial 90 finished with value: -0.5675848959814518 and parameters: {'beta': 0.05181818941453341, 'init_prev': 0.017604127320651544, 'n_contacts': 5, 'rand_seed': 935995}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.740 s

[I 2025-02-14 05:58:24,281] Trial 91 finished with value: 0.021987357424855025 and parameters: {'beta': 0.10113340967153214, 'init_prev': 0.029547066726050977, 'n_contacts': 5, 'rand_seed': 949145}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.737 s

[I 2025-02-14 05:58:24,991] Trial 92 finished with value: -0.2103922521198282 and parameters: {'beta': 0.09963308265328698, 'init_prev': 0.015345395180435631, 'n_contacts': 5, 'rand_seed': 962755}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.759 s

[I 2025-02-14 05:58:25,230] Trial 93 finished with value: -0.5611024280265263 and parameters: {'beta': 0.06687801511723575, 'init_prev': 0.015266258992421042, 'n_contacts': 5, 'rand_seed': 924031}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.752 s

[I 2025-02-14 05:58:25,929] Trial 94 finished with value: -0.5561026613005221 and parameters: {'beta': 0.0767710552546615, 'init_prev': 0.01094676524059421, 'n_contacts': 5, 'rand_seed': 911557}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.732 s

[I 2025-02-14 05:58:26,149] Trial 95 finished with value: -0.4677404510151619 and parameters: {'beta': 0.04684871868327455, 'init_prev': 0.011268854701392241, 'n_contacts': 4, 'rand_seed': 981013}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.727 s

[I 2025-02-14 05:58:26,844] Trial 96 finished with value: -0.4734147417824611 and parameters: {'beta': 0.04594709484785002, 'init_prev': 0.013894700925278555, 'n_contacts': 4, 'rand_seed': 860329}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.731 s

[I 2025-02-14 05:58:27,069] Trial 97 finished with value: -0.4541474805545193 and parameters: {'beta': 0.03904903759505303, 'init_prev': 0.017640728149432527, 'n_contacts': 4, 'rand_seed': 686456}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.758 s

[I 2025-02-14 05:58:27,785] Trial 98 finished with value: -0.5486955220090516 and parameters: {'beta': 0.03831346114745416, 'init_prev': 0.021871410906193864, 'n_contacts': 6, 'rand_seed': 782850}. Best is trial 72 with value: -0.5749499034003444.

Elapsed time: 0.702 s

[I 2025-02-14 05:58:27,940] Trial 99 finished with value: -0.47607878824730165 and parameters: {'beta': 0.05516198387658676, 'init_prev': 0.01983108690957649, 'n_contacts': 7, 'rand_seed': 789018}. Best is trial 72 with value: -0.5749499034003444.

Making results structure...
Processed 100 trials; 0 failed
Best pars: {'beta': 0.0549476616412528, 'init_prev': 0.0277422478690141, 'n_contacts': 5, 'rand_seed': 666971}
Removed existing calibration file starsim_calibration.db

Let’s look at the best parameters that were found. Note that the rand_seed was selected at random, but the other parameters are meaningful.

calib.best_pars

{'beta': 0.0549476616412528,
 'init_prev': 0.0277422478690141,
 'n_contacts': 5,
 'rand_seed': 666971}

Once the calibration is complete, we can compare the guess values to the best values found by calling check_fit.

# Confirm - Note the comparison is here configured over n_reps=15 replicates
sc.printcyan('\nConfirming fit...')

# Increase replicates to 15 for more representative results when running check_fit
calib.build_kw['n_reps'] = 15

calib.check_fit(do_plot=False)

Confirming fit...

Checking fit...
Elapsed time: 4.96 s
Fit with original pars: -0.38944292453422913
Fit with best-fit pars: -0.5712977009791536
✓ Calibration improved fit -0.38944292453422913 --> -0.5712977009791536

True

After calling check_fit, we can plot the results. This first plot shows the Normal likelihood distributions from each of the 15 simulations we did in check_fit as the colored lines. The vertical dashed line is located at the real (expected) data. Top row is the “guess” values and the bottom row is the new “best” parameters. We want the vertical dashed line to cross the Gaussians at high points, representing high likelihood.

calib.plot();

Another way to plot the results is via bootstrapping. Here we repeatedly choose 15 from the n_reps=15 simulations (with replacement), compute the average (or sum for some components), and repeatedly calculate the mean. We then plot the distribution of means, and hope it lands near the vertical dashed lines representing the real data.

calib.plot(bootstrap=True); # Pass bootstrap=True to produce this plot

We can view some plots of the final fitted results. Whereas the two plots above were from the check_fit, running both “guess” and “best” parameters, here we make make new simulations to visualize the results.

g = calib.plot_final(); # Run the model for build_kw['n_reps'] = 15 replicates
for ax in g.axes: # Fix the date formatting
    ax.xaxis.set_major_formatter(mdates.ConciseDateFormatter(ax.xaxis.get_major_locator()))

Elapsed time: 2.38 s

Optuna has lots of diagnostic plots that we can explore. Possible plots include:

• plot_contour

• plot_edf

• plot_hypervolume_history

• plot_intermediate_values

• plot_optimization_history

• plot_parallel_coordinate

• plot_param_importances

• plot_pareto_front

• plot_rank

• plot_slice

• plot_terminator_improvement

• plot_timeline

Here are some examples:

calib.plot_optuna('plot_optimization_history'); # Plot the optimization history

/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-starsim/envs/latest/lib/python3.11/site-packages/starsim/calibration.py:436: ExperimentalWarning: plot_optimization_history is experimental (supported from v2.2.0). The interface can change in the future.
  fig = getattr(vis, method)(self.study)

calib.plot_optuna('plot_contour');

/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-starsim/envs/latest/lib/python3.11/site-packages/starsim/calibration.py:436: ExperimentalWarning: plot_contour is experimental (supported from v2.2.0). The interface can change in the future.
  fig = getattr(vis, method)(self.study)
[W 2025-02-14 05:58:46,883] Output figures of this Matplotlib-based `plot_contour` function would be different from those of the Plotly-based `plot_contour`.

calib.plot_optuna('plot_param_importances');

Could not run plot_param_importances: Tried to import 'sklearn' but failed. Please make sure that the package is installed correctly to use this feature. Actual error: No module named 'sklearn'.

/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-starsim/envs/latest/lib/python3.11/site-packages/starsim/calibration.py:436: ExperimentalWarning: plot_param_importances is experimental (supported from v2.2.0). The interface can change in the future.
  fig = getattr(vis, method)(self.study)

If you choose not to use components, you can always create your own mismatch function, as in the following example:

my_data = (ss.date('2020-01-12'), 0.13)

def eval(sim, expected):
    # Compute the squared error at one point in time.
    # expected will contain my_data in this example due to eval_kw
    date, p = expected
    if not isinstance(sim, ss.MultiSim):
        sim = ss.MultiSim(sims=[sim])

    ret = 0
    for s in sim.sims:
        ind = np.searchsorted(s.results.timevec, date, side='left')
        prev = s.results.sir.prevalence[ind]
        ret += (prev - p)**2
    return ret


# Define the calibration parameters
calib_pars = dict(
    beta = dict(low=0.01, high=0.30, guess=0.15, suggest_type='suggest_float', log=True),
)

# Make the sim and data
sim = make_sim()

# Make the calibration
calib = ss.Calibration(
    calib_pars = calib_pars,
    sim = sim,
    build_fn = build_sim,
    build_kw = dict(n_reps=2), # Two reps per point
    reseed = True,
    eval_fn = eval, # Will call my_function(msim, eval_kwargs)
    eval_kw = dict(expected=my_data), # Will call eval(sim, **eval_kw)
    total_trials = 10,
    n_workers = None, # None indicates to use all available CPUs
    die = True,
    debug = debug,
)

# Perform the calibration
sc.printcyan('\nPeforming calibration...')
calib.calibrate()

# Check
calib.check_fit()

Peforming calibration...
Removed existing calibration file starsim_calibration.db
sqlite:///starsim_calibration.db

[I 2025-02-14 05:58:49,492] A new study created in RDB with name: starsim_calibration

Elapsed time: 0.462 s
Elapsed time: 0.458 s

[I 2025-02-14 05:58:50,212] Trial 0 finished with value: 0.030135273527771893 and parameters: {'beta': 0.02424906028651729, 'rand_seed': 463699}. Best is trial 0 with value: 0.030135273527771893.
[I 2025-02-14 05:58:50,230] Trial 1 finished with value: 0.0313755649150238 and parameters: {'beta': 0.018441522453043358, 'rand_seed': 955605}. Best is trial 0 with value: 0.030135273527771893.

Elapsed time: 0.452 s

[I 2025-02-14 05:58:50,753] Trial 2 finished with value: 0.2051161716426711 and parameters: {'beta': 0.23595338020204482, 'rand_seed': 225749}. Best is trial 0 with value: 0.030135273527771893.

Elapsed time: 0.486 s

[I 2025-02-14 05:58:50,838] Trial 3 finished with value: 0.001011147257163561 and parameters: {'beta': 0.06063294033714838, 'rand_seed': 673742}. Best is trial 3 with value: 0.001011147257163561.

Elapsed time: 0.447 s

[I 2025-02-14 05:58:51,297] Trial 4 finished with value: 0.22979766394640444 and parameters: {'beta': 0.22575163276761123, 'rand_seed': 752947}. Best is trial 3 with value: 0.001011147257163561.

Elapsed time: 0.449 s

[I 2025-02-14 05:58:51,380] Trial 5 finished with value: 0.18845433019136051 and parameters: {'beta': 0.2609600775798452, 'rand_seed': 927846}. Best is trial 3 with value: 0.001011147257163561.

Elapsed time: 0.473 s

[I 2025-02-14 05:58:51,863] Trial 6 finished with value: 0.1240514789183259 and parameters: {'beta': 0.10472509166369227, 'rand_seed': 375213}. Best is trial 3 with value: 0.001011147257163561.

Elapsed time: 0.456 s

[I 2025-02-14 05:58:51,932] Trial 7 finished with value: 0.2131988012778525 and parameters: {'beta': 0.23380160057578603, 'rand_seed': 448623}. Best is trial 3 with value: 0.001011147257163561.

Elapsed time: 0.487 s

[I 2025-02-14 05:58:52,443] Trial 8 finished with value: 0.01340191201153353 and parameters: {'beta': 0.07726792954705192, 'rand_seed': 680446}. Best is trial 3 with value: 0.001011147257163561.

Elapsed time: 0.459 s

[I 2025-02-14 05:58:52,474] Trial 9 finished with value: 0.35485042032341885 and parameters: {'beta': 0.22702141866225437, 'rand_seed': 725865}. Best is trial 3 with value: 0.001011147257163561.

Making results structure...
Processed 10 trials; 0 failed
Best pars: {'beta': 0.06063294033714838, 'rand_seed': 673742}
Removed existing calibration file starsim_calibration.db

Checking fit...
Elapsed time: 0.706 s
Fit with original pars: 0.45052830548072875
Fit with best-fit pars: 0.006480500000000001
✓ Calibration improved fit 0.45052830548072875 --> 0.006480500000000001

True

For more, take a look at test_calibration.py in the tests directory.