Operational best practices

As part of developing a calibration workflow, there are some technical aspects that are important to get right. Ideally, these points can be determined by the re-identify known parameters step listed above. What gives the best results? Specific considerations here include the following.

Develop workflows

Calibration is not the endpoint of most analyses! Develop a workflow in which the results of calibration are archived while keeping sufficient version information to know where the results came from. You will likely need to run more simulations than is reasonable on a laptop. Aim to make the workflow as simple as possible, as complex workflows are often fragile.

Version management

To be able to later reproduce your results, ensure your workflow stores or otherwise records versions of everything. We often prefer to store inputs (e.g. GitHub hashes and local deltas) as opposed to outputs (big files from simulations). Docker is typically not worth the effort in our experience, but is a worthwhile consideration. Consider an experiment tracking tool, like MLFlow.

Initialization and burn-in

As related to edge case handling, when does the simulation start? How many and which (age/sex/factors) agents are initially infected? Do those initial infections clear quickly, risking burn-out? Have you initialized any immunity into the initial population? Is this an outbreak scenario?

Number of agents

How many agents are included per simulation? Larger numbers of agents, \(N\), results in reduced variance typically on order of \(1/N\), but takes longer to evaluate. Consider how to balance the tradeoff between reduced variance and simulation time.

Number of replicates

How many replicates are included of each parameter configuration? More replicates takes time away from searching other parameter configurations, but too few replicates could result in noise that often confounds calibration algorithms.

Performance

Is there anything you can do to make the simulations faster? Remove an analyzer? Truncate the run time to the last data point? Profile the code and look for "hot lines" that are frequently executed and good candidates for changes to improve performance.