emodpy documentation¶
emodpy is a collection of Python scripts and utilities created to streamline user interactions with EMOD and idmtools. Additional functionality for interacting with EMOD is provided in the emod_api package and idmtools packages.
See idmtools for a diagram showing how idmtools and each of the related packages are used in an end-to-end workflow using EMOD as the disease transmission model.
Installation¶
You can install emodpy in two different ways. If you intend to use emodpy as IDM builds it, follow the instructions in Basic installation. However, if you intend to modify the emodpy source code to add new functionality, follow the instructions in Developer installation. Whichever installation method you choose, the prerequisites are the same.
Prerequisites¶
Python 3.9 64-bit (https://www.python.org/downloads/release)
Python virtual environments
Python virtual environments enable you to isolate your Python environments from one another and give you the option to run multiple versions of Python on the same computer. When using a virtual environment, you can indicate the version of Python you want to use and the packages you want to install, which will remain separate from other Python environments. You may use
virtualenv, which requires a separate installation, butvenvis recommended and included with Python 3.3+.
Basic installation¶
Follow the steps below if you will use idmtools to run and analyze simulations, but will not make source code changes.
Open a command prompt and create a virtual environment in any directory you choose. The command below names the environment “emodpy”, but you may use any desired name, and any available path you prefer:
python -m venv /path/to/venv/root/emodpy
Activate the virtual environment:
On Windows, enter the following:
\path\to\venv\root\emodpy\Scripts\activate
On Linux, enter the following:
source /path/to/venv/root/emodpy/bin/activate
Install idmtools packages.
pip install emodpy --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
(It’s strongly recommended that you edit your pip.ini or pip.conf so you don’t have to specificy –index-url.)
Verify installation by doing a test import:
python -c 'import emodpy'
When you are finished, deactivate the virtual environment by entering the following at a command prompt:
deactivate
Developer installation¶
Follow the steps below if you will make changes to the idmtools source code to add new functionality.
Install idmtools¶
Install a Git client such as Git Bash or the Git GUI.
Open a command prompt and clone the idmtools GitHub repository to a local directory using the following command:
git clone https://github.com/InstituteforDiseaseModeling/emodpy-idmtools.git To work from the latest approved code, work from the "master" branch. To work from the latest code under active development, work from the "dev" branch.
Open a command prompt and create a virtual environment in any directory you choose. The command below names the environment “emodpy”, but you may use any desired name:
python -m venv emodpy
Activate the virtual environment:
On Windows, enter the following:
emodpy\Scripts\activate
On Linux, enter the following:
source emodpy/bin/activate
In the base directory of the cloned GitHub repository, run the setup script.
On Windows, enter the following:
pip install py-make pymake setup-dev
On Linux, enter the following:
make setup-dev
To verify that idmtools is installed, enter the following command:
emodpy --help
You should see a list of available cookie cutter projects and command-line options.
Run tests¶
If you want to run tests on the code, do the following. You can add new tests to the GitHub repository and they will be run using the same commands. Note that COMPS access is generally restricted to IDM employees.
Login to COMPS by navigating to the idmtools root directory and entering the following at a command prompt:
python dev_scripts\create_auth_token_args.py --comps_url https://comps2.idmod.org --username yourcomps_user --password yourcomps_password
If you are running the local platform with the nightly idmtools build, enter the following to log in to Docker:
docker login idm-docker-staging.packages.idmod.org
Navigate to the directory containing the code you want to test, such as the root directory or a subdirectory like emodpy_platform_comps, enter the following command:
pymake test-all
Create simulations¶
Overview¶
Creating a simulation generally consists of 4 parts: - Creating the model configuration - Defining the demographics (and migration) - Building the campaign - Configuring your reports
Model Configuration¶
Model configuration starts with the schema which is provided along with the model binary in a emod-disease module, e.g., emod-measles. The emod-api module, a dependency of emodpy, provides the functionality to go from schema to configuration. You will pass a config builder function to the emod_task.from_default2 function here in emodpy.
Demographics¶
Model configuration almost always includes some kind of specification of the demographics you want to model, even if it’s just the number of people in your sim. You will do this in a demographics builder function which also gets passed to emod_task.from_default2. A working demographics configuration can be created from emod-api.demographics functionality, but most emodpy-disease modules have a demographics submodule with disease-specific capabilities.
Campaign¶
After specifying the details of your disease and the people in your simulation, you’ll soon want to start adding interventions. This is done in a campaign builder function, often called build_camp, but can be named what you want. You will also pass this function to emod_task.from_default2(). Your campaign will be built up from calls to intervention-specific functions in your emodpy-disease.interventions submodule. Though emod-api.interventions has some very simple starter functionality, like the ability to seed an outbreak which is important.
A campaign will consist of scheduled campaign events and/or triggered campaign events.
A scheduled campaign event results in an intervention being distributed to people (or nodes) at a given time. And possibly repeated.
A triggered campaign event listens for triggers or signals and distributes an intervention to individuals at that time.
Triggered Campaigns¶
Triggered campaigns are a very powerful and popular way to build campaigns in EMOD. This is very much like a publish-subscribe (pub-sub) architecture for those familiar with that, or the signals and slots design in Qt. There are two kinds of signals (sometimes called events or triggers) that are published (or broadcast): model signals and campaign signals. Model signals are built right into the code and occur on events like births, birthdays, deaths, new infections, etc. The exact list varies depending on the particular disease you are working with. For a complete list, see the documentation for your emodpy-disease.intervention submodule. Campaign signals are published based on your campaign setup. Some interventions have default signals, like perhaps ‘PositiveTestResult’, but users can use ad-hoc signals that are previously unknown to the model. Any published signal can then be listened to by another campaign event. So for example you can distribute a diagnostic which listens for a ‘NewInfection’ signal from the model, and publishes a ‘Tested_Positive_For_Pox’ signal in the case of a positive test (which is going to be very likely if it’s responding to NewInfection signals but let’s skip that for now). Then you can distribute a therapeutic intervention that listens for your ‘Tested_Positive_For_Pox’ signal. These would all be done with the TriggeredCampaignEvent function in emod-api.interventions.common.
Reports¶
Once you have your disease model configured, your human demographics set up, your campaign details added, you’ll want to get some outputs using built-in or plugin reporters. Some disease models rely on single, catch-all report or output file, while other diseases have a veritable panoply of reporters. These are configured very much like the model itself, where the schema providers parameters with default values and you will set specific parameters. Some complex reports have helper functions in emodpy-disease submodules.
Create input files¶
Run simulations¶
Calibrate simulations¶
Parameter sweeps and model iteration¶
Parameter sweeps for model calibration¶
(more info) For more information on model calibration, see Calibrate simulations.
Parameter sweeps and stochasticity¶
With a stochastic model (such as EMOD), it is especially important to utilize parameter sweeps, not only for calibration to data or parameter selection, but to fully explore the stochasticity in output. Single model runs may appear to provide good fits to data, but variation will arise and multiple runs are necessary to determine the appropriate range of parameter values necessary to achieve desired outcomes. Multiple iterations of a single set of parameter values should be run to determine trends in simulation output: a single simulation output could provide results that are due to random chance.
How to do parameter sweeps¶
Introduction to analyzers¶
Output reports¶
Serialization¶
API reference¶
emodpy package¶
Subpackages¶
emodpy.analyzers package¶
Submodules¶
emodpy.analyzers.adult_vectors_analyzer module¶
- class emodpy.analyzers.adult_vectors_analyzer.AdultVectorsAnalyzer(name='hi')[source]¶
Bases:
IAnalyzer- initialize()[source]¶
Call once after the analyzer has been added to the
AnalyzeManager.Add everything depending on the working directory or unique ID here instead of in __init__.
- map(data: Any, item: IItem) Any[source]¶
In parallel for each simulation/work item, consume raw data from filenames and emit selected data.
- Parameters:
data – A dictionary associating filename with content for simulation data.
item –
IItemobject that the passed data is associated with.
- Returns:
Selected data for the given simulation/work item.
emodpy.analyzers.population_analyzer module¶
- class emodpy.analyzers.population_analyzer.PopulationAnalyzer(name='idm')[source]¶
Bases:
IAnalyzer- initialize()[source]¶
Call once after the analyzer has been added to the
AnalyzeManager.Add everything depending on the working directory or unique ID here instead of in __init__.
- map(data: Any, item: IItem) Any[source]¶
In parallel for each simulation/work item, consume raw data from filenames and emit selected data.
- Parameters:
data – A dictionary associating filename with content for simulation data.
item –
IItemobject that the passed data is associated with.
- Returns:
Selected data for the given simulation/work item.
emodpy.analyzers.timeseries_analyzer module¶
- class emodpy.analyzers.timeseries_analyzer.TimeseriesAnalyzer(filenames=['output/InsetChart.json'], channels=('Statistical Population', 'Infectious Population', 'Infected', 'Waning Population'), save_output=True)[source]¶
Bases:
IAnalyzer- data_group_names = ['group', 'sim_id', 'channel']¶
- ordered_levels = ['channel', 'group', 'sim_id']¶
- output_file = 'timeseries.csv'¶
- initialize()[source]¶
Call once after the analyzer has been added to the
AnalyzeManager.Add everything depending on the working directory or unique ID here instead of in __init__.
- filter(simulation)[source]¶
Decide whether the analyzer should process a simulation/work item.
- Parameters:
item – An
IItemto be considered for processing with this analyzer.- Returns:
A Boolean indicating whether simulation/work item should be analyzed by this analyzer.
- map(data, simulation)[source]¶
In parallel for each simulation/work item, consume raw data from filenames and emit selected data.
- Parameters:
data – A dictionary associating filename with content for simulation data.
item –
IItemobject that the passed data is associated with.
- Returns:
Selected data for the given simulation/work item.
emodpy.defaults package¶
Subpackages¶
Submodules¶
emodpy.generic package¶
Submodules¶
emodpy.generic.serialization module¶
- emodpy.generic.serialization.enable_serialization(task: EMODTask, use_absolute_times: bool = False)[source]¶
Enable serialization etierh by TIME or TIMESTEP based on use_absolute_times :param task: Task to enable :param use_absolute_times: When true, Serialization_Type will be set to TIME, otherwise it will be set to :param *TIMESTEP*:
Returns:
- emodpy.generic.serialization.add_serialization_timesteps(task: EMODTask, timesteps: List[int], end_at_final: bool = False, use_absolute_times: bool = False)[source]¶
Serialize the population of this simulation at specified time steps.
If the simulation is run on multiple cores, multiple files will be created.
- Parameters:
task (EMODTask) – An EMODSimulation
timesteps (List[int]) – Array of integers representing the time steps to use
end_at_final (bool) – False means set the simulation duration such that the last serialized_population file ends the simulation. NOTE- may not work if time step size is not 1
use_absolute_times (bool) – False means the method will define simulation times instead of time steps see documentation on Serialization_Type for details
- Returns:
None
emodpy.interventions package¶
Submodules¶
emodpy.interventions.emod_empty_campaign module¶
- class emodpy.interventions.emod_empty_campaign.EMODEmptyCampaign[source]¶
Bases:
IEMODDefault- static campaign() EMODCampaign[source]¶
emodpy.reporters package¶
Submodules¶
emodpy.reporters.base module¶
- class emodpy.reporters.base.CustomReporter(name: str | None = None, Enabled: bool = True, Reports: list = <factory>, dll_file: str | None = None)[source]¶
Bases:
BaseReporterThis class represents a custom reporter. - name: Name that will be added to the custom_reports.json file and should match the DLL’s class name - Enabled: True/False to enable/disable the reporter - Reports: Default section present in the custom_reports.json file allowing to configure the reporter - dll_file: Filename of the dll containing the reporter. This file will be searched in the dll folder specified by the user on the EMODTask.reporters.
- class emodpy.reporters.base.BuiltInReporter(class_name: str = None, parameters: dict = <factory>, Enabled: bool = True, Pretty_Format: bool = True)[source]¶
Bases:
BaseReporter
- class emodpy.reporters.base.Reporters(relative_path='reporter_plugins')[source]¶
Bases:
InputFilesList- property json¶
- property empty¶
- add_dll(dll_path: str)[source]¶
Add a dll file from a path
- Parameters:
dll_path – Path to file
Returns:
- add_dll_folder(dll_folder: str)[source]¶
Add all the dll files from a folder
- Parameters:
dll_folder – Folder to add the dll file from
Returns:
- read_custom_reports_file(custom_reports_path, extra_classes=[]) NoReturn[source]¶
Read from a custom reporter file
- Parameters:
custom_reports_path – The custom reports file to add(single file).
emodpy.reporters.builtin module¶
- class emodpy.reporters.builtin.ReportNodeDemographics(class_name: str = 'ReportNodeDemographics', parameters: dict = <factory>, Enabled: bool = True, Pretty_Format: bool = True, Stratify_By_Gender: bool = False, Age_Bins: list = <factory>)[source]¶
Bases:
BuiltInReporter
emodpy.reporters.custom module¶
- class emodpy.reporters.custom.ReportAgeAtInfectionHistogramPlugin(name: str = 'ReportPluginAgeAtInfectionHistogram', Enabled: bool = True, Reports: list = <factory>, dll_file: str = 'libReportAgeAtInfectionHistogram_plugin.dll', age_bins: list = <factory>, interval_years: int = <factory>)[source]¶
Bases:
CustomReporter
- class emodpy.reporters.custom.ReportHumanMigrationTracking(name: str = 'ReportHumanMigrationTracking', Enabled: bool = True, Reports: list = <factory>, dll_file: str = 'libhumanmigrationtracking.dll')[source]¶
Bases:
CustomReporterThe human migration tracking report is a CSV-formatted report that provides details about human travel during simulations. The finished report will provide one line for each surviving individual that migrates during the simulation. There are no special parameters that need to be configured to generate the report.
- class emodpy.reporters.custom.ReportNodeDemographics(name: str = 'ReportNodeDemographics', Enabled: bool = True, Reports: list = <factory>, dll_file: str = 'libReportNodeDemographics.dll')[source]¶
Bases:
CustomReporterThe node demographics report is a CSV-formatted report that provides population information stratified by node. For each time step, the report will collect data on each node and age bin.
- configure_report(age_bins=None, ip_key_to_collect='', stratify_by_gender=1)[source]¶
Creates the report and sets up the parameters.
- Parameters:
age_bins – The Age Bins (in years) to aggregate within and report; an empty array implies ‘do not stratify by age.
ip_key_to_collect – The name of the IndividualProperty key to stratify by; an empty string implies ‘do not stratify by IP.’
stratify_by_gender – Set to true (1) to stratify by gender; a value of 0 will not stratify by gender.
- Returns:
Nothing
- class emodpy.reporters.custom.ReportEventCounter(name: str = 'ReportEventCounter', Enabled: bool = True, Reports: list = <factory>, dll_file: str = 'libreporteventcounter.dll')[source]¶
Bases:
CustomReporterThe event counter report is a JSON-formatted file that keeps track of how many of each event types occurs during a time step. The report produced is similar to the InsetChart.json channel report, where there is one channel for each event defined in the configuration file (config.json).
- configure_report(duration_days=10000, event_trigger_list=None, nodes=None, report_description='', start_day=0)[source]¶
Create the report and set up the parameters.
- Parameters:
duration_days – The duration of simulation days over which to report events.
event_trigger_list – The list of event triggers for the events included in the report.
nodes – The list of nodes in which to track the events, setting it to None or [] tracks all nodes.
report_description – Name of the report (it augments the filename of the report). If multiple CSV reports are being generated, this allows the user to distinguish one report from another.
start_day – The day to start collecting data for the report.
- Returns:
Nothing
Submodules¶
emodpy.bamboo module¶
emodpy.bamboo_api_utils module¶
- class emodpy.bamboo_api_utils.BambooConnection[source]¶
Bases:
objectBamboo API config and basic functionality/connectivity wrapper.
Automatically probes the most likely endpoint locations (with and without https, with and without port numbers).
Important functions:
login: logs into the bamboo api, caches the login token so you don’t have to pass creds for every req. in a session
get_bamboo_api_url: translate a relative API URL into a fully qualified URL
normalize_url: detect whether a URL is relative or not, translate relative URLs to fully qualified ones
make_get_request: makes a request to the specified API url, adds some convenient error and login handling
download_file: downloads a file from the specified artifacts url to a location on disk
- property server: str¶
str: Keeps track of a single instance of the server base url. (e.g. http://idm-bamboo:8085)
- property session_cookie: <module 'requests.cookies' from '/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-emodpy/envs/v1.22.0/lib/python3.9/site-packages/requests/cookies.py'>¶
str: Automatically load and instance the login session cookie jar.
- get_server_url(ssl: bool = True, useport: bool = False) str[source]¶
Get a particular variant of the server url w/ or w/o ssl and port (e.g. False/False -> http://idm-bamboo)
- find_server() str[source]¶
Explore all possible server urls, return the first one found to exist.
- Returns:
server url
- Return type:
- url_exists(url: str) bool[source]¶
Try a simple get request given an endpoint url, return whether it was successful (code 200).
- property session_cookie_filename: str¶
File where bamboo session cookie is stored.
- Returns:
fully qualified file path of session cookie file
- Return type:
- load_session_cookie() <module 'requests.cookies' from '/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-emodpy/envs/v1.22.0/lib/python3.9/site-packages/requests/cookies.py'>[source]¶
Load api login session cookies from disk.
- Returns:
session cookie jar
- Return type:
requests.cookies
- write_session_cookie(cookies: <module 'requests.cookies' from '/home/docs/checkouts/readthedocs.org/user_builds/institute-for-disease-modeling-emodpy/envs/v1.22.0/lib/python3.9/site-packages/requests/cookies.py'>)[source]¶
Write post-login cookies for session to disk.
- get_bamboo_api_url(relative_url: str, json: bool = False, params: dict = {}) str[source]¶
Get fully qualified bamboo api url from a relative url w/ given json mode and appending all parameters.
- save_credentials(username: str, password: str)[source]¶
Save bamboo api login credentials using keyring.
- Parameters:
username (str) – bamboo api login username (e.g. somebody@idmod.org)
password (str) – bamboo api login password
- ensure_logged_in()[source]¶
Check if a login session exists using saved cookies, if not login using keyring stored creds.
- login_session_exists() bool[source]¶
Test whether an existing session cookie exists and an active login session exists.
- Returns:
whether an active login session exists
- Return type:
- login(username: str | None = None, password=None) bool[source]¶
Login to the bamboo api. If username or password are not provided, use stored credentials from keyring.
- Parameters:
username (str) – bamboo api login username (e.g. somebody@idmod.org)
password (str) – bamboo api login password
- Returns:
success/failure
- Return type:
- normalize_url(url: str) str[source]¶
Determine whether a url is relative or fully qualified, translate relative urls to fully qualified versions.
- make_get_request(url: str, retries: int = 3) Response[source]¶
Make a get request against the bamboo server.
- Parameters:
url (str) – relative or fully qualified url
- Returns:
request object returned from requests.get()
- Return type:
requests.Response
- make_api_get_request(relative_url: str, json: bool = False, params: dict = {}) Response[source]¶
Translate relative api url to the fully qualified bamboo api url, make a get request against it.
- Parameters:
- Returns:
request object returned from requests.get()
- Return type:
requests.Response
- class emodpy.bamboo_api_utils.BuildInfo[source]¶
Bases:
objectA collection of methods for getting data on build results.
- classmethod build_passed(plan_key: str, build_num: int) bool[source]¶
Determine whether a given build succeeded or not.
- static successful_build_result(result) bool[source]¶
Analyze a build result json object and determine if it corresponds to a successful build
- Parameters:
result – json build result
- Returns:
whether the build was successful
- Return type:
- static get_build_info(plan_key: str, index: int)[source]¶
Retrieve the build info in json format for a given build plan with a relative index (0=latest)
- classmethod get_latest_successful_build(plan_key: str, scheduled_only: bool = True, max_iterations: int = 100)[source]¶
Find the latest successful build within the last max_iterations builds for a given plan.
- Parameters:
- Returns:
- tuple containing:
build_num (str): build number of last successful build build_info: json data structure of build info for that build
- Return type:
(tuple)
- class emodpy.bamboo_api_utils.BuildArtifacts[source]¶
Bases:
objectA collection of methods for finding and interacting with build artifacts.
- ERADICATION_EXE = 'Eradication.exe'¶
- SCHEMA_JSON = 'schema.json'¶
- REPORTER_PLUGINS = 'Reporter-Plugins'¶
- classmethod find_artifacts_by_name(plan_key: str, build_num: int, artifact: str) list[source]¶
Find all urls for files of an artifact of a given name for a specific build.
- classmethod find_artifacts(plan_key: str, build_num: int, artifact_list: list) list[source]¶
Find all urls for files of a list of artifacts for a specific build.
- classmethod find_build_essential_artifacts(plan_key: str, build_num: int) list[source]¶
Find all ‘build essential’ artifact urls (Eradication, schema, reporters) for a specific build
- classmethod find_all_artifacts(plan_key: str, build_num: int) list[source]¶
Find all artifact urls (Eradication, schema, reporters) for a specific build
- classmethod find_all_artifact_names(plan_key: str, build_num: int) list[source]¶
Find all artifact names (e.g. ‘Eradication.exe’) for a specific build (can be plugged into find_artifacts() to get actual urls that can be downloaded)
- classmethod download_artifact_to_file(plan_key: str, build_num: int, artifact, destination: str) list[source]¶
Download files found for a named artifact to the filepath provided.
Additional files found will be downloaded as _2, _3, _4, etc. For example, if there are 3 files for ‘Eradication.exe’ the first will be Eradication.exe, the second will be Eradication_2.exe, the third Eradication_3.exe.
- Parameters:
- Returns:
list of local filenames of files that have been downloaded
- Return type:
- classmethod download_artifacts_to_path(plan_key: str, build_num: int, artifact, destination_path: str) list[source]¶
Download all the files for a given artifact and build to a specific folder, using their original filenames.
- Parameters:
- Returns:
list of local filenames of files that have been downloaded
- Return type:
- classmethod download_latest_good_Eradication_exe(plan_key: str, destination: str) str[source]¶
Find the latest successful build for a specified plan, download the Eradication.exe artifact to a specified path.
- classmethod download_latest_good_schema_json(plan_key: str, destination: str) str[source]¶
Find the latest successful build for a specified plan, download the schema.json artifact to a specified path.
- classmethod download_eradication_exe(plan_key: str, build_num: str, destination: str) str[source]¶
Download Eradication.exe artifact from a specific build.
- classmethod make_exe_executable(file_path: str)[source]¶
On linux change the file permissions on a binary to make it executable
- Parameters:
file_path (str) – binary file to mark as executable
- classmethod download_schema_json(plan_key: str, build_num: str, destination: str) str[source]¶
Download schema.json artifact from a specific build.
- class emodpy.bamboo_api_utils.BuildPlans[source]¶
Bases:
objectCollection of methods for getting information on build plans.
emodpy.collections_utils module¶
- emodpy.collections_utils.cut_iterable_to(obj: Iterable, to: int) Tuple[List | Mapping, int][source]¶
Cut an iterable to a certain length.
- Parameters:
obj – The iterable to cut.
to – The number of elements to return.
- Returns:
A list or dictionary (depending on the type of object) of elements and the remaining elements in the original list or dictionary.
- emodpy.collections_utils.deep_get(d, key, default: callable | None = None, getter: callable | None = None, sep: str = '.')[source]¶
emodpy.emod_campaign module¶
- class emodpy.emod_campaign.EMODCampaign(name='Campaign', events=None, use_defaults=True, **kwargs)[source]¶
Bases:
objectClass representing an EMOD Campaign. It contains: - events: a list of events for the given campaign - name: campaign name - use_defaults: EMOD flag to use defaults for unspecified parameters - extra_parameters: parameters set by the user that will be added to the campaign JSON
- property json¶
Property to transform the object in JSON
- static load_from_file(filename: str) object[source]¶
Load a campaign from a JSON file.
- Parameters:
filename – Path to the campaign file
Returns: an initialized EMODCampaign instance
- static load_from_dict(data: Dict) object[source]¶
Create a campaign object from a dict. :param data: The dictionary containing the data
Returns: an initialized EMODCampaign instance
- get_events_at(timestep: int) List[Dict][source]¶
Get a list of events happening at the specified timestep. Does not take into account recurrence and only consider start timestep. :param timestep: selected timestep
Returns: list of events
- get_events_with_name(name: str) List[Dict][source]¶
Get a list of events with the given name. This search is based on the Event_Name key of events. :param name: Name of the events
Returns: list of events
emodpy.emod_file module¶
- class emodpy.emod_file.InputFilesList(relative_path=None)[source]¶
Bases:
AssetCollection
- class emodpy.emod_file.MigrationTypes(value)[source]¶
Bases:
EnumAn enumeration.
- LOCAL = 'Local'¶
- AIR = 'Air'¶
- FAMILY = 'Family'¶
- REGIONAL = 'Regional'¶
- SEA = 'Sea'¶
- class emodpy.emod_file.MigrationModel(value)[source]¶
Bases:
EnumAn enumeration.
- NO_MIGRATION = 'NO_MIGRATION'¶
- FIXED_RATE_MIGRATION = 'FIXED_RATE_MIGRATION'¶
- class emodpy.emod_file.MigrationPattern(value)[source]¶
Bases:
EnumAn enumeration.
- RANDOM_WALK_DIFFUSION = 'RANDOM_WALK_DIFFUSION'¶
- SINGLE_ROUND_TRIPS = 'SINGLE_ROUND_TRIPS'¶
- WAYPOINTS_HOME = 'WAYPOINTS_HOME'¶
- class emodpy.emod_file.MigrationFiles(relative_path=None)[source]¶
Bases:
InputFilesList- enable_migration()[source]¶
Enables migration and sets the pattern if defined. If there are not other other parameters, it also set Enable_Migration_Heterogeneity to 0
- update_migration_pattern(migration_pattern: MigrationPattern, **kwargs) NoReturn[source]¶
Update migration pattern
- Parameters:
migration_pattern – Migration Pattern to use
**kwargs –
- Returns:
NoReturn
- add_migration_from_file(migration_type: MigrationTypes, file_path: str, multiplier: float = 1)[source]¶
Add migration info from a file
- Parameters:
migration_type – Type of migration
file_path – Path to file
multiplier – Multiplier
Returns:
- set_task_config(task: EMODTask)[source]¶
Update the task with the migration configuration
- Parameters:
task – Task to update
Returns:
- merge_with(mf: MigrationFiles, left_precedence: bool = True) NoReturn[source]¶
Merge migration file with other Migration file
- Parameters:
mf – Other migration file to merge with
left_precedence – Does the current object have precedence or the other object?
Returns:
- class emodpy.emod_file.DemographicsFiles(relative_path=None)[source]¶
Bases:
InputFilesList- set_task_config(task: EMODTask, extend: bool = False)[source]¶
Set the simulation level config. If extend is true, the demographics files are appended to the list :param task: :param extend:
Returns:
- class emodpy.emod_file.ClimateFileType(value)[source]¶
Bases:
EnumAn enumeration.
- AIR_TEMPERATURE = 'Air_Temperature'¶
- LAND_TEMPERATURE = 'Land_Temperature'¶
- RELATIVE_HUMIDITY = 'Relative_Humidity'¶
- RAINFALL = 'Rainfall'¶
- class emodpy.emod_file.ClimateModel(value)[source]¶
Bases:
EnumAn enumeration.
- CLIMATE_OFF = 'CLIMATE_OFF'¶
- CLIMATE_CONSTANT = 'CLIMATE_CONSTANT'¶
- CLIMATE_KOPPEN = 'CLIMATE_KOPPEN'¶
- CLIMATE_BY_DATA = 'CLIMATE_BY_DATA'¶
- class emodpy.emod_file.ClimateFiles[source]¶
Bases:
InputFilesList- set_task_config(task: EMODTask)[source]¶
Set the task Config. Set all the correct files for the climate.
- Parameters:
task – Task to config
emodpy.emod_task module¶
- emodpy.emod_task.dev_mode = False¶
Note that these 3 functions could be member functions of EMODTask but Python modules are already pretty good at being ‘static classes’.
- emodpy.emod_task.add_ep4_from_path(task, ep4_path='EP4')[source]¶
Add embedded Python scripts from a given path.
- class emodpy.emod_task.EMODTask(command: str | ~idmtools.entities.command_line.CommandLine = <property object>, platform_requirements: ~typing.Set[~idmtools.entities.platform_requirements.PlatformRequirements] = <factory>, _ITask__pre_creation_hooks: ~typing.List[~typing.Callable[[~idmtools.entities.simulation.Simulation | ~idmtools.entities.iworkflow_item.IWorkflowItem, ~idmtools.entities.iplatform.IPlatform], ~typing.NoReturn]] = <factory>, _ITask__post_creation_hooks: ~typing.List[~typing.Callable[[~idmtools.entities.simulation.Simulation | ~idmtools.entities.iworkflow_item.IWorkflowItem, ~idmtools.entities.iplatform.IPlatform], ~typing.NoReturn]] = <factory>, common_assets: ~idmtools.assets.asset_collection.AssetCollection = <factory>, transient_assets: ~idmtools.assets.asset_collection.AssetCollection = <factory>, eradication_path: str | None = None, demographics: ~emodpy.emod_file.DemographicsFiles = <factory>, migrations: ~emodpy.emod_file.MigrationFiles = <factory>, reporters: ~emodpy.reporters.base.Reporters = <factory>, climate: ~emodpy.emod_file.ClimateFiles = <factory>, config: dict = <factory>, config_file_name: str = 'config.json', campaign: ~emodpy.emod_campaign.EMODCampaign = <factory>, simulation_demographics: ~emodpy.emod_file.DemographicsFiles = <factory>, simulation_migrations: ~emodpy.emod_file.MigrationFiles = <factory>, use_embedded_python: bool = True, py_path_list: list = <factory>, is_linux: bool = False, implicit_configs: list = <factory>, sif_filename: str | None = None)[source]¶
Bases:
ITaskEMODTask allows easy running and configuration of EMOD Experiments and Simulations
- demographics: DemographicsFiles¶
Common Demographics
- migrations: MigrationFiles¶
Common Migrations
- climate: ClimateFiles¶
Common Climate
- campaign: EMODCampaign¶
Campaign configuration
- simulation_demographics: DemographicsFiles¶
Simulation level demographics such as overlays
- simulation_migrations: MigrationFiles¶
Simulation level migrations
- sif_path = None¶
- create_campaign_from_callback(builder, params=None)[source]¶
- Parameters:
write_campaign (str) – if not None, the path to write the campaign to
- handle_implicit_configs()[source]¶
Execute the implicit config functions created by the demographics builder.
- classmethod from_default2(eradication_path, schema_path, param_custom_cb=None, config_path='config.json', campaign_builder=None, ep4_custom_cb=<function default_ep4_fn>, demog_builder=None, plugin_report=None, serial_pop_files=None, write_default_config=None, ep4_path=None, **kwargs) EMODTask[source]¶
Create a task from emod-api Defaults
- Parameters:
eradication_path – Path to Eradication binary.
schema_path – Path to schema.json.
param_custom_cb – Function that sets parameters for config.
campaign_builder – Function that builds the campaign.
ep4_custom_cb – Function that sets EP4 assets. There are 4 options for specificying EP4 scripts: 1) Set ep4_custom_cb=None. This just says “don’t even attempt to use EP4 scripts”. Not the default. 2) All defaults. This uses the built-in ep4 scripts (inside emodpy module) which does some standard pre- and post-processing. You can’t edit these scripts. 3) Set the (new) ep4_path to your local path where your custom scripts are. Leave out ep4_custom_cb so it uses the default function (but with your path). 4) Power mode where you set ep4_custom_cb to your own function which gives you all the power. Probably don’t need this.
demog_builder – Function that builds the demographics configuration and optional migration configuration.
plugin_report – Custom reports file.
serial_pop_files – Input “.dtk” serialized population files.
config_path – Optional filename for the generated config.json, if you don’t like that name.
write_default_config – Set to true if you want to have the default_config.json written locally for inspection.
ep4_path – See ep4_custom_cb section.
- Returns:
EMODTask
- classmethod from_files(eradication_path=None, config_path=None, campaign_path=None, demographics_paths=None, ep4_path=None, custom_reports_path=None, asset_path=None, **kwargs)[source]¶
Load custom EMOD files when creating
EMODTask.- Parameters:
asset_path – If an asset path is passed, the climate, dlls, and migrations will be searched there
eradication_path – The eradication.exe path.
config_path – The custom configuration file.
campaign_path – The custom campaign file.
demographics_paths – The custom demographics files (single file or a list).
custom_reports_path – Custom reports file
Returns: An initialized experiment
- load_files(config_path=None, campaign_path=None, custom_reports_path=None, demographics_paths=None, asset_path=None) NoReturn[source]¶
Load files in the experiment/base_simulation.
- Parameters:
asset_path – Path to find assets
config_path – Configuration file path
campaign_path – Campaign file path
demographics_paths – Demographics file path
custom_reports_path – Path for the custom reports file
- pre_creation(parent: Simulation | IWorkflowItem, platform: IPlatform)[source]¶
Call before a task is executed. This ensures our configuration is properly done
- add_py_path(path_to_add) NoReturn[source]¶
Add path to list of python paths prepended to sys.path in embedded interpreter
Returns:
- gather_common_assets() AssetCollection[source]¶
Gather Experiment Level Assets Returns:
- gather_transient_assets() AssetCollection[source]¶
Gather assets that are per simulation Returns:
- copy_simulation(base_simulation: Simulation) Simulation[source]¶
Called when making copies of a simulation.
Here we deep copy parts of the simulation to ensure we don’t accidentally update objects :param base_simulation: Base Simulation
Returns:
- set_parameter(name: str, value: any) dict[source]¶
Set a value in the EMOD config.json file. This will be deprecated in the future in favour of emod_api.config.
- Parameters:
name – Name of parameter to set
value – Value to set
- Returns:
Tags to set
- static set_parameter_sweep_callback(simulation: Simulation, param: str, value: Any) Dict[str, Any][source]¶
Convenience callback for sweeps
- Parameters:
simulation – Simulation we are updating
param – Parameter
value – Value
- Returns:
Tags to set on simulation
- classmethod set_parameter_partial(parameter: str)[source]¶
Convenience callback for sweeps
- Parameters:
parameter – Parameter to set
Returns:
- get_parameter(name: str, default: Any | None = None)[source]¶
Get a parameter in the simulation.
- Parameters:
name – The name of the parameter.
default – Optional, the default value.
- Returns:
The value of the parameter.
- update_parameters(params)[source]¶
Bulk update the configuration parameter values. This will be deprecated in the future in favour of emod_api.config.
- Parameters:
params – A dictionary with new values.
- Returns:
None
- reload_from_simulation(simulation: Simulation)[source]¶
Optional hook that is called when loading simulations from a platform.
- classmethod get_file_from_comps(exp_id, filename)[source]¶
Get file or files from COMPS. Retrieve all files named <filename> in experiment <exp_id> and put them in a local directory called exp_id. On linux, this is under “latest_experiment”. This function will eventually be added to pyCOMPS.
- classmethod cache_experiment_metadata_in_sql(exp_id, optional_data_files=None)[source]¶
Create local sqlite database of experiment metadata, plus optional data from post-proc file. Tags will be column names.
- Parameters:
exp_id – ID of experiment.
optional_data_files – List of filenames (not path) of downloaded files containing single value post-processed on server.
- Returns:
None.
- class emodpy.emod_task.EMODTaskSpecification[source]¶
Bases:
TaskSpecification- get(configuration: dict) EMODTask[source]¶
Return an EMODTask object using provided configuration :param configuration: Configuration for Task
- Returns:
EMODTask for configuration
emodpy.utils module¶
- class emodpy.utils.EradicationPlatformExtension(value)[source]¶
Bases:
EnumAn enumeration.
- LINUX = ''¶
- Windows = '.exe'¶
- class emodpy.utils.EradicationBambooBuilds(value)[source]¶
Bases:
EnumAn enumeration.
- GENERIC_LINUX = 'DTKGENCI-SCONSLNXGEN'¶
- GENERIC_WIN = 'DTKGENCI-SCONSWINGEN'¶
- GENERIC = 'DTKGENCI-SCONSLNXGEN'¶
- TBHIV_LINUX = 'DTKTBHIVCI-SCONSRELLNXTBHIV'¶
- TBHIV_WIN = 'DTKTBHIVCI-SCONSWINTBHIV'¶
- TBHIV = 'DTKTBHIVCI-SCONSRELLNXTBHIV'¶
- MALARIA_LINUX = 'DTKMALCI-SCONSLNXMAL'¶
- MALARIA_WIN = 'DTKMALCI-SCONSWINMAL'¶
- MALARIA = 'DTKMALCI-SCONSLNXMAL'¶
- HIV_LINUX = 'DTKHIVCI-SCONSRELLNXHIV'¶
- HIV_WIN = 'DTKHIVCI-RELWINHIV'¶
- HIV = 'DTKHIVCI-SCONSRELLNXHIV'¶
- DENGUE_LINUX = 'DTKDENGCI-SCONSRELLNX'¶
- DENGUE_WIN = 'DTKDENGCI-VSRELWINALL'¶
- DENGUE = 'DTKDENGCI-SCONSRELLNX'¶
- FP_LINUX = 'DTKFPCI-SCONSRELLNX'¶
- FP_WIN = 'DTKFPCI-SCONSWINFP'¶
- FP = 'DTKFPCI-SCONSRELLNX'¶
- TYPHOID_LINUX = 'DTKTYPHCI-SCONSRELLNX'¶
- TYPHOID_WIN = 'DTKTYPHCI-SCONSWINENV'¶
- TYPHOID = 'DTKTYPHCI-SCONSRELLNX'¶
- EMOD_RELEASE = 'EMODREL-SCONSRELLNX'¶
- RELEASE = 'DTKREL-SCONSRELLNX'¶
- class emodpy.utils.BambooArtifact(value)[source]¶
Bases:
FlagAn enumeration.
- ERADICATION = 1¶
- SCHEMA = 2¶
- PLUGINS = 4¶
- ALL = 7¶
- emodpy.utils.get_github_eradication_url(version: str, extension: EradicationPlatformExtension = EradicationPlatformExtension.LINUX) str[source]¶
Get the github eradication url for specified release
- Parameters:
version – Release to fetch
extension – Optional extensions. Defaults to Linux(None)
- Returns:
Url of eradication release
- emodpy.utils.save_bamboo_credentials(username, password)[source]¶
Save bamboo api login credentials using keyring.
- Parameters:
username (str) – bamboo api login username (e.g. somebody@idmod.org)
password (str) – bamboo api login password
- emodpy.utils.bamboo_api_login()[source]¶
Automatically login to bamboo, prompt for credentials if none are cached or there’s no login session.
- emodpy.utils.download_bamboo_artifacts(plan_key: str, build_num: str | None = None, scheduled_builds_only: bool = True, artifact: BambooArtifact = BambooArtifact.ERADICATION, out_path: str | None = None) list[source]¶
Downloads artifact(s) for a DTK Bamboo build plan to the specified path
- Parameters:
plan_key (str) –
build_num (str) –
scheduled_builds_only (bool) –
artifact (BambooArtifact) –
out_path (str) – Output path to save file (default to current directory)
- Returns:
Returns list of downloaded files on filesystem
- emodpy.utils.download_latest_bamboo(plan: EradicationBambooBuilds, scheduled_builds_only: bool = True, out_path: str | None = None) str[source]¶
Downloads the Eradication binary for the latest successful build for a Bamboo Plan to specified path. Exists for backward compatibility, just a pass-thru to download_latest_eradication().
- Parameters:
plan – Bamboo Plan key. for supported build
out_path – Output path to save file (default to current directory)
- Returns:
Returns local filename of downloaded file
- emodpy.utils.download_latest_eradication(plan: EradicationBambooBuilds, scheduled_builds_only: bool = True, out_path: str | None = None) str[source]¶
Downloads the Eradication binary for the latest successful build for a Bamboo Plan to specified path.
- Parameters:
plan – Bamboo Plan key. for supported build
out_path – Output path to save file (default to current directory)
- Returns:
Returns local filename of downloaded file
- emodpy.utils.download_latest_reporters(plan: EradicationBambooBuilds, scheduled_builds_only: bool = True, out_path: str | None = None) list[source]¶
Downloads the reporter plugins for the latest successful build for a Bamboo Plan to specified path.
- Parameters:
plan – Bamboo Plan key. for supported build
out_path – Output path to save file (default to current directory)
- Returns:
Returns list of local filenames of downloaded files
- emodpy.utils.download_latest_schema(plan: EradicationBambooBuilds, scheduled_builds_only: bool = True, out_path: str | None = None) str[source]¶
Downloads the schema.json for the latest successful build for a Bamboo Plan to specified path.
- Parameters:
plan – Bamboo Plan key. for supported build
out_path – Output path to save file (default to current directory)
- Returns:
Returns local filename of downloaded file
- emodpy.utils.download_eradication(url: str, cache_path: str = None, spinner=None)[source]¶
Downloads Eradication binary
Useful for downloading binaries from Bamboo or Github
- Parameters:
url – Url to binary
cache_path – Optional output directory
spinner – Spinner object
- Returns:
Full path to output file
Frequently asked questions¶
As you get started with emodpy, you may have questions. The most common questions are answered below. If you are using a disease-specific emodpy package, see the FAQs from that package for additional guidance. For questions related to functionality in related packages, see the following documentation:
Frequently asked questions for EMOD
Frequently asked questions for idmtools
Frequently asked questions for emod-api
Why does emodpy download a new Eradication binary each time I run?¶
emodpy is designed to work much like a web browser: when you go to a website, the browser downloads html, png, and other files. If you visit the page again, it downloads them again so you always have the most current files. We want emodpy to work in much the same way. When you run simulations, emodpy will download the latest tested binary, schema, and supporting files that from the relevant EMOD ongoing branch.
However, if you need the stability of working from an older version, you can
pass a Bamboo build number to emodpy.bamboo.get_model_files() to
download that build instead. If you want to manually add a binary and and
corresponding schema in the downloads directory to use, comment out the call
to emodpy.bamboo.get_model_files() and nothing new will be
downloaded.
What is the purpose of manifest.py?¶
The manifest.py file contains all of your input and output paths in a single location. It also includes the path where model binaries (and associated schema) are downloaded to and uploaded from. Although you may ignore these files, it can be helpful to reference the schema for parameter information and have access to the binary itself.
I want to load a demographics.json file, not create one programmatically.¶
Okay, but be aware that one of the benefits of emodpy and emod-api is that you get guaranteed consistency between demographics and configuration parameters to meet all interdependencies. However, if you want to use a raw demographics.json that you are very confident in, you can open that in your demographics builder. For example:
def build_demog():
import emod_api.demographics.Demographics as Demographics
demog = Demographics.from_file( "demographics.json" )
return demog
What happens if I don’t connect to the VPN?¶
You must be connected to the IDM VPN to access Bamboo and download the
Eradication binaries (including plug-ins and schema). As an alternative, comment
out the call to emodpy.bamboo.get_model_files() in the code
and run the following (where “emod-disease” can be “emodpy-hiv”, “emodpy-malaria”,
or “emod-measles”:
pip install emod-disease --upgrade
python -m emod-disease.bootstrap
The model files will be in a subdirectory called “stash.”
Why are the example.py scripts read from the bottom?¶
A Python script’s “main” block, which is also the entry point to the run script, appears at the end so that all the functions in the script have been parsed and are available. It is a common convention to structure the call flow bottom-up because of that.
My simulation failed on COMPS but I didn’t get an error until then¶
The OS of the requested Bamboo build plan and the OS of the target platform need to match. For example, if your target platform is Calculon, the default, you’ll have to request a Linux build from Bamboo. There are no protections at this time (nor planned) to catch such misconfigurations.
How do I make my sim run inside a custom environment (on COMPS) for the first time?¶
There are 3 small steps for this:
Add a line of code:
task.set_sif( manifest.sif )
to your main Python script, after the task variable has been created.
Add a line to your manifest.py file like:
sif = "emod_sif.id"
Create a new file called ‘emod_sif.id’ – just match the name you used in step 2 – and put an asset collection id in it. At time of writing, this is the tested SIF asset id in the Calculon environment for running EMOD with Python3.9 and emod-api pre-installed:
f1e6b032-47b7-ec11-a9f6-9440c9be2c51
You can find a quasi-catalog of available SIF ids here: https://github.com/InstituteforDiseaseModeling/singularity_image_files/tree/master/emod.
Note that you can of course just do this in one step, and add a line of code to your script like:
task.set_sif( "f1e6b032-47b7-ec11-a9f6-9440c9be2c51" )
But it’s much preferred to follow the above pattern so that future changes to use another SIF can be isolated to the resource file.
Is there a Singularity Image File that lets me run a version of the model that’s built against Python3.9?¶
Yes. Assuming you already have a task.set_sif() call in your script, replace the current contents of your dtk_centos.id (or emod_sif.id) file with the following: f1e6b032-47b7-ec11-a9f6-9440c9be2c51. You may want to back up your existing version of that file.
What if I need a new or different SIF with a different custom environment?¶
Anyone is free to create SIFs for themselves and use those. COMPS can build SIFs for you provided a ‘recipe’ – .def file. There are people at IDM who can do it on their desktops. Bear in mind Singularity really only installs on Linux.
How do I specify the number of cores to use on the cluster?¶
num_cores is an undocumented kwargs argument to Platform. What that means is if you already have a script with a line like:
platform = Platform( "SLURM" )``
you would change it to something like:
platform = Platform( "SLURM", num_cores=4 )
to run with 4 cores.
What does “DTK” stand for?¶
Disease Transmission Kernel. This was the early internal name of EMOD.
What is a “parameter sweep”?¶
When the docs refer to a “parameter sweep”, it usually means an experiment consisting of a multiple simulations where almost all the input values are the same except for a single parameter. The parameter being swept will have different values across a range, possibly the min to the max, but any range of interest to the modeler. Parameter sweeps can be very useful for just learning the sensitivity of a given parameter, or as a form of manual calibration. A “1-D parameter sweep” is where you just sweep over a single parameter. You can also do “2-D parameter sweeps”, where you sweep over two parameters at once, and so on. But these of course require more simulations and fancier visualization.
A special kind of parameter sweep is sweeping over Run_Number which is the random number seed. This kind of sweep gives you a sense of the model to general stochasticity, given your other inputs.
You can sweep over config, demographics, or campaign parameters.
Is there any place where I can see which parameters are taken from distributions and what type of distributions are they?¶
Any parameter that is being set from a distribution will have the distribution type in the name. E.g., Base_Infectivity_Gaussian_Mean tells you that this value is being drawn from a Gaussian distribution. If you don’t see any distribution name in the parameter name, it’s just fixed at that parameter value.
Glossary¶
The following terms describe both the features and functionality of the emodpy software, as well as information relevant to using emodpy.
- asset collection¶
The set of specific input files (such as input parameters, weather or migration data, or other configuration settings) required for running a simulation.
- assets¶
See asset collection.
- builder¶
TBD
- experiment¶
A collection of multiple simulations, typically sent to an HPC.
- high-performance computing (HPC)¶
The use of parallel processing for running advanced applications efficiently, reliably, and quickly.
- task¶
TBD
- template¶
TBD
Changelog¶
1.1.0¶
Additional Changes¶
Bugs¶
#0011 - task with simulation level demographics not work
#0012 - How to add custom simulation tags from task?
#0040 - examples- emod_model- serialization- 03_parameter_reload getting wrong campaign
#0042 - We should make EMODSir default work with eradication
#0043 - Wired campaign format error
#0044 - Examples- create_sims_pre_and_post_process.py should import config_update_parameters correctly
#0055 - Creation of campaign.json will fail in AC in COMPS - cannot overwrite AC files
#0059 - EmodTask.pre_post_process should be renamed
#0069 - Fix create_serialized_sims_reload and create_sims_from_default_run_analyzer examples
#0072 - custom_reports.json - not all of them have “enabled”, but code assumes they do
#0073 - Climate_Model should be set to whatever it is set in config.json when from_files is used.
#0075 - custom_reports: when reading my ReportNodeDemographics report, one of the parameters is not read in
Developer/Test¶
Documentation¶
Feature Request¶
Models¶
Platforms¶
#0021 - SSMT Build as part of GithubActions
User Experience¶
1.2.0¶
Additional Changes¶
#0091 - Eradication.exe can’t consume emodpy_covid installed in a virtual environment (Windows)
Bugs¶
Documentation¶
#0060 - Help with repro: dtk_pre_process executed twice before simulation attempted