T7 - Calibration¶
Tutorial 2 demonstrated how to run the model and plot the outputs. But it’s entirely possible that the model outputs won’t look like the data for the country that you wish to model. The default parameter values included in HPVsim are intended as points of departure to be iteratively refined via calibration. The process of model calibration involves finding the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.
Click here to open an interactive version of this notebook.
Data types supported by HPVsim¶
Data on HPV and cervical disease comes in many different formats. When using HPVsim, the goal is typically to produce population-level estimates of epidemic outputs like: - age-specific incidence of cancer or high-grade lesions in one or more years; - number of cases of cancer or high-grade lesions reported in one or more years; - HPV prevalence over time; - lifetime incidence of HPV; - the distribution of genotypes in detected cases of cancer/high-grade lesions; - sexual behavior metrics like the average age at first marriage, duration of relationships, or number of lifetime partners.
After running HPVsim, estimates all of these variables are included within the results dictionary. To plot them alongside data, the easiest method is to use the Calibration object.
The Calibration object¶
Calibration objects contain the following ingredients: - an hpv.Sim() instance with details of the model configuration; - two lists of parameters to vary, one for parameters that vary by genotype and one for those that don’t; - dataframes that hold the calibration targets, which are typically added as csv files; - a list of any additional results to plot; - settings that are passed to the Optuna package[LINK], an open source hyperparameter optimization framework that automates calibration
for HPVsim.
We have included Optuna as a built-in calibration option as we have found that it works reasonably well, but it is also possible to use other methods; we will discuss this a little further down.
The example below illustrates the general idea of calibration, and can be adapted for different use cases:
[1]:
# Import HPVsim
import hpvsim as hpv
# Configure a simulation with some parameters
pars = dict(n_agents=10e3, start=1980, end=2020, dt=0.25, location='nigeria')
sim = hpv.Sim(pars)
# Specify some parameters to adjust during calibration.
# The parameters in the calib_pars dictionary don't vary by genotype,
# whereas those in the genotype_pars dictionary do. Both kinds are
# given in the order [best, lower_bound, upper_bound].
calib_pars = dict(
beta=[0.05, 0.010, 0.20],
dur_transformed=dict(par1=[5, 3, 10]),
)
genotype_pars = dict(
hpv16=dict(
sev_rate=[0.5, 0.2, 1.0],
dur_episomal=dict(par1=[6, 4, 12])
),
hpv18=dict(
sev_rate=[0.5, 0.2, 1.0],
dur_episomal=dict(par1=[6, 4, 12])
)
)
# List the datafiles that contain data that we wish to compare the model to:
datafiles=['nigeria_cancer_cases.csv',
'nigeria_cancer_types.csv']
# List extra results that we don't have data on, but wish to include in the
# calibration object so we can plot them.
results_to_plot = ['cancer_incidence', 'asr_cancer_incidence']
# Create the calibration object, run it, and plot the results
calib = hpv.Calibration(
sim,
calib_pars=calib_pars,
genotype_pars=genotype_pars,
extra_sim_results=results_to_plot,
datafiles=datafiles,
total_trials=3, n_workers=1
)
calib.calibrate(die=True)
calib.plot(res_to_plot=4);
HPVsim 1.1.1 (2023-03-01) — © 2023 by IDM
Loading location-specific demographic data for "nigeria"
Loading location-specific demographic data for "nigeria"
Initializing sim with 10000 agents
Loading location-specific data for "nigeria"
/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-hpvsim/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
[I 2023-03-10 03:22:17,335] A new study created in RDB with name: hpvsim_calibration
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration hpvsim_calibration.db
Loading location-specific demographic data for "nigeria"
Running 1980.0 ( 0/164) (0.00 s) ———————————————————— 1%
Running 1982.5 (10/164) (0.28 s) •——————————————————— 7%
Running 1985.0 (20/164) (0.53 s) ••—————————————————— 13%
Running 1987.5 (30/164) (0.78 s) •••————————————————— 19%
Running 1990.0 (40/164) (1.04 s) •••••——————————————— 25%
Running 1992.5 (50/164) (1.30 s) ••••••—————————————— 31%
Running 1995.0 (60/164) (1.58 s) •••••••————————————— 37%
Running 1997.5 (70/164) (1.85 s) ••••••••———————————— 43%
Running 2000.0 (80/164) (2.12 s) •••••••••——————————— 49%
Running 2002.5 (90/164) (2.41 s) •••••••••••————————— 55%
Running 2005.0 (100/164) (2.70 s) ••••••••••••———————— 62%
Running 2007.5 (110/164) (3.01 s) •••••••••••••——————— 68%
Running 2010.0 (120/164) (3.32 s) ••••••••••••••—————— 74%
Running 2012.5 (130/164) (3.64 s) •••••••••••••••————— 80%
Running 2015.0 (140/164) (3.99 s) •••••••••••••••••——— 86%
Running 2017.5 (150/164) (4.34 s) ••••••••••••••••••—— 92%
[I 2023-03-10 03:22:22,910] Trial 0 finished with value: 7.5781968824768 and parameters: {'hpv16_sev_rate': 0.734051946780452, 'hpv16_dur_episomal_par1': 8.21713774532916, 'hpv18_sev_rate': 0.26331557461695027, 'hpv18_dur_episomal_par1': 8.664558301716921, 'beta': 0.012789926208581544, 'dur_transformed_par1': 6.0240912979036665}. Best is trial 0 with value: 7.5781968824768.
Running 2020.0 (160/164) (4.68 s) •••••••••••••••••••— 98%
Simulation summary:
86,193 infections
0 dysplasias
0 pre-cins
0 cin1s
28,731 cin2s
0 cin3s
57,462 cins
718 cancers
0 cancer detections
1,437 cancer deaths
0 detected cancer deaths
86,193 reinfections
0 reactivations
616,958,400 number susceptible
354,830 number infectious
1,437 number with inactive infection
205,755,248 number with no cellular changes
354,830 number with episomal infection
718 number with transformation
1,437 number with cancer
356,266 number infected
356,266 number with abnormal cells
0 number with latent infection
346,210 number with precin
405,109 number with cin1
80,447 number with cin2
66,800 number with cin3
109,178 number with carcinoma in situ
661,534 number with detectable dysplasia
0 number with detected cancer
0 number screened
0 number treated for precancerous lesions
0 number treated for cancer
0 number vaccinated
0 number given therapeutic vaccine
0.00 hpv incidence (/100)
0 cin1 incidence (/100,000)
0 cin2 incidence (/100,000)
0 cin3 incidence (/100,000)
0 dysplasia incidence (/100,000)
1 cancer incidence (/100,000)
7,434,184 births
2,355,954 other deaths
-129,290 migration
1 age-adjusted cervical cancer incidence (/100,000)
0 age-adjusted cervical cancer mortality
0 newly vaccinated
0 cumulative number vaccinated
0 new doses
0 cumulative doses
0 new therapeutic vaccine doses
0 newly received therapeutic vaccine
0 cumulative therapeutic vaccine doses
0 total received therapeutic vaccine
0 new screens
0 newly screened
0 new cin treatments
0 newly treated for cins
0 new cancer treatments
0 newly treated for cancer
0 cumulative screens
0 cumulative number screened
0 cumulative cin treatments
0 cumulative number treated for cins
0 cumulative cancer treatments
0 cumulative number treated for cancer
0 detected cancer incidence (/100,000)
1 cancer mortality
205,755,248 number alive
0 crude death rate
0 crude birth rate
0.06 hpv prevalence (/100)
0 pre-cin prevalence (/100,000)
0 cin1 prevalence (/100,000)
0 cin2 prevalence (/100,000)
0 cin3 prevalence (/100,000)
Loading location-specific demographic data for "nigeria"
Running 1980.0 ( 0/164) (0.00 s) ———————————————————— 1%
Running 1982.5 (10/164) (0.30 s) •——————————————————— 7%
Running 1985.0 (20/164) (0.61 s) ••—————————————————— 13%
Running 1987.5 (30/164) (0.93 s) •••————————————————— 19%
Running 1990.0 (40/164) (1.27 s) •••••——————————————— 25%
Running 1992.5 (50/164) (1.61 s) ••••••—————————————— 31%
Running 1995.0 (60/164) (1.96 s) •••••••————————————— 37%
Running 1997.5 (70/164) (2.35 s) ••••••••———————————— 43%
Running 2000.0 (80/164) (2.77 s) •••••••••——————————— 49%
Running 2002.5 (90/164) (3.18 s) •••••••••••————————— 55%
Running 2005.0 (100/164) (3.61 s) ••••••••••••———————— 62%
Running 2007.5 (110/164) (4.06 s) •••••••••••••——————— 68%
Running 2010.0 (120/164) (4.51 s) ••••••••••••••—————— 74%
Running 2012.5 (130/164) (5.02 s) •••••••••••••••————— 80%
Running 2015.0 (140/164) (5.52 s) •••••••••••••••••——— 86%
Running 2017.5 (150/164) (6.05 s) ••••••••••••••••••—— 92%
Running 2020.0 (160/164) (6.60 s) •••••••••••••••••••— 98%
[I 2023-03-10 03:22:30,426] Trial 1 finished with value: 7.993771512906608 and parameters: {'hpv16_sev_rate': 0.7492124595976735, 'hpv16_dur_episomal_par1': 9.133467405051878, 'hpv18_sev_rate': 0.2596561405211299, 'hpv18_dur_episomal_par1': 10.186958606544792, 'beta': 0.11716451474213739, 'dur_transformed_par1': 5.376659558789292}. Best is trial 0 with value: 7.5781968824768.
Simulation summary:
69,803,037 infections
0 dysplasias
0 pre-cins
4,807,438 cin1s
1,137,035 cin2s
471,191 cin3s
14,635,643 cins
21,548 cancers
0 cancer detections
10,056 cancer deaths
0 detected cancer deaths
58,821,273 reinfections
0 reactivations
488,623,520 number susceptible
75,647,672 number infectious
193,217 number with inactive infection
205,935,520 number with no cellular changes
75,647,672 number with episomal infection
177,415 number with transformation
193,217 number with cancer
75,840,896 number infected
75,840,896 number with abnormal cells
0 number with latent infection
35,777,456 number with precin
31,183,346 number with cin1
5,961,712 number with cin2
4,334,093 number with cin3
1,491,146 number with carcinoma in situ
38,705,160 number with detectable dysplasia
0 number with detected cancer
0 number screened
0 number treated for precancerous lesions
0 number treated for cancer
0 number vaccinated
0 number given therapeutic vaccine
4.76 hpv incidence (/100)
0 cin1 incidence (/100,000)
0 cin2 incidence (/100,000)
0 cin3 incidence (/100,000)
0 dysplasia incidence (/100,000)
21 cancer incidence (/100,000)
7,434,184 births
2,207,988 other deaths
-244,215 migration
31 age-adjusted cervical cancer incidence (/100,000)
0 age-adjusted cervical cancer mortality
0 newly vaccinated
0 cumulative number vaccinated
0 new doses
0 cumulative doses
0 new therapeutic vaccine doses
0 newly received therapeutic vaccine
0 cumulative therapeutic vaccine doses
0 total received therapeutic vaccine
0 new screens
0 newly screened
0 new cin treatments
0 newly treated for cins
0 new cancer treatments
0 newly treated for cancer
0 cumulative screens
0 cumulative number screened
0 cumulative cin treatments
0 cumulative number treated for cins
0 cumulative cancer treatments
0 cumulative number treated for cancer
0 detected cancer incidence (/100,000)
10 cancer mortality
205,935,520 number alive
0 crude death rate
0 crude birth rate
12.24 hpv prevalence (/100)
0 pre-cin prevalence (/100,000)
0 cin1 prevalence (/100,000)
0 cin2 prevalence (/100,000)
0 cin3 prevalence (/100,000)
Loading location-specific demographic data for "nigeria"
Running 1980.0 ( 0/164) (0.00 s) ———————————————————— 1%
Running 1982.5 (10/164) (0.29 s) •——————————————————— 7%
Running 1985.0 (20/164) (0.57 s) ••—————————————————— 13%
Running 1987.5 (30/164) (0.86 s) •••————————————————— 19%
Running 1990.0 (40/164) (1.17 s) •••••——————————————— 25%
Running 1992.5 (50/164) (1.48 s) ••••••—————————————— 31%
Running 1995.0 (60/164) (1.80 s) •••••••————————————— 37%
Running 1997.5 (70/164) (2.14 s) ••••••••———————————— 43%
Running 2000.0 (80/164) (2.51 s) •••••••••——————————— 49%
Running 2002.5 (90/164) (2.90 s) •••••••••••————————— 55%
Running 2005.0 (100/164) (3.28 s) ••••••••••••———————— 62%
Running 2007.5 (110/164) (3.69 s) •••••••••••••——————— 68%
Running 2010.0 (120/164) (4.10 s) ••••••••••••••—————— 74%
Running 2012.5 (130/164) (4.55 s) •••••••••••••••————— 80%
Running 2015.0 (140/164) (5.01 s) •••••••••••••••••——— 86%
Running 2017.5 (150/164) (5.48 s) ••••••••••••••••••—— 92%
Running 2020.0 (160/164) (5.96 s) •••••••••••••••••••— 98%
[I 2023-03-10 03:22:37,264] Trial 2 finished with value: 6.545380406479723 and parameters: {'hpv16_sev_rate': 0.5262070436046669, 'hpv16_dur_episomal_par1': 11.081371043611643, 'hpv18_sev_rate': 0.3718207482147955, 'hpv18_dur_episomal_par1': 11.778806911281652, 'beta': 0.044169781318392755, 'dur_transformed_par1': 5.4910148824324}. Best is trial 2 with value: 6.545380406479723.
Simulation summary:
21,105,899 infections
0 dysplasias
0 pre-cins
2,273,352 cin1s
591,143 cin2s
293,058 cin3s
7,966,428 cins
11,492 cancers
0 cancer detections
7,183 cancer deaths
0 detected cancer deaths
16,404,047 reinfections
0 reactivations
572,073,152 number susceptible
41,497,832 number infectious
81,165 number with inactive infection
205,842,144 number with no cellular changes
41,497,832 number with episomal infection
63,209 number with transformation
81,165 number with cancer
41,578,992 number infected
41,578,992 number with abnormal cells
0 number with latent infection
14,915,055 number with precin
17,140,282 number with cin1
3,168,327 number with cin2
1,927,141 number with cin3
731,926 number with carcinoma in situ
21,922,584 number with detectable dysplasia
0 number with detected cancer
0 number screened
0 number treated for precancerous lesions
0 number treated for cancer
0 number vaccinated
0 number given therapeutic vaccine
1.23 hpv incidence (/100)
0 cin1 incidence (/100,000)
0 cin2 incidence (/100,000)
0 cin3 incidence (/100,000)
0 dysplasia incidence (/100,000)
11 cancer incidence (/100,000)
7,434,184 births
2,319,322 other deaths
-186,752 migration
16 age-adjusted cervical cancer incidence (/100,000)
0 age-adjusted cervical cancer mortality
0 newly vaccinated
0 cumulative number vaccinated
0 new doses
0 cumulative doses
0 new therapeutic vaccine doses
0 newly received therapeutic vaccine
0 cumulative therapeutic vaccine doses
0 total received therapeutic vaccine
0 new screens
0 newly screened
0 new cin treatments
0 newly treated for cins
0 new cancer treatments
0 newly treated for cancer
0 cumulative screens
0 cumulative number screened
0 cumulative cin treatments
0 cumulative number treated for cins
0 cumulative cancer treatments
0 cumulative number treated for cancer
0 detected cancer incidence (/100,000)
7 cancer mortality
205,842,144 number alive
0 crude death rate
0 crude birth rate
6.72 hpv prevalence (/100)
0 pre-cin prevalence (/100,000)
0 cin1 prevalence (/100,000)
0 cin2 prevalence (/100,000)
0 cin3 prevalence (/100,000)
Loading saved results...
Removed temporary file tmp_calibration_00000.obj
Loaded trial 0
Removed temporary file tmp_calibration_00001.obj
Loaded trial 1
Removed temporary file tmp_calibration_00002.obj
Loaded trial 2
Making results structure...
Processed 3 trials; 0 failed
Deleted study hpvsim_calibration in sqlite:///hpvsim_calibration.db
Removed existing calibration hpvsim_calibration.db
This isn’t a great fit yet! In general, it will probably be necessary to run many more trials that the 3 we ran here. Moreover, careful consideration should be given to the parameters that you want to adjust during calibration. In HPVsim we have taken the approach that any parameter can be adjusted. As we learn more about which parameters make most sense to calibrate, we will add details here. We would also enourage users to share their experiences with calibration and parameter searches.
[ ]:
[ ]:
[ ]:
[ ]:
[ ]:
[ ]:
[ ]:
[ ]: