Parameter DB: Store EMOD parameter info

The EMOD parameter database is located at http://paramdb.internal.idm.ctr/. You must be connected to the IDM VPN to access the paramDB. The database stores all EMOD parameter information (config, campaign, and demographics) for all sim types, such as min/max/default values, extended descriptions, JSON examples, etc. We run queries against the DB to export CSV files for the documentation, which Sphinx builds into tables. Because campaign parameters can appear in multiple classes and configuration parameters can appear in multiple categories, this provides a way for us to single source the parameter information to ensure consistency and accuracy.

Do not depend entirely on tickets or communication of EMOD changes from developers to make DB updates! Rather, review the schema for supported sim types and current parameters to verify changes in the branches you are documenting. You can either build the schema yourself (see Generate a list of all available parameters (a schema)) or download a schema from Bamboo at http://idmppapp09.internal.idm.ctr:8085/artifact/DTKMASTER.

We recommend using WinMerge to view the difference between a known past version of the schema and a current version of the schema. The emod/csv folder is only for parameter DB output; do not make changes in this folder.

Note

We previously had an agreement to document only parameters and features in the master branch of DtkTrunk, not in unmerged working branches. This has been modified recently to include documenting parameters in the following ongoing branches: malaria-ongoing, hiv-ongoing, and generic-ongoing. However, given that this agreement is new as of spring 2021, the documentation is not yet current with the ongoing branches.

The paramDB currently cannot support ongoing branches. To document ongoing branches in the interim until the paramDB has been updated, manually make changes to the CSV files. These changes will then be ingested by the paramDB. Remember to make changes in all CSV files in which a given parameter appears to avoid merge conflicts later.

Warning

Do not use Excel to manually edit CSV files. Excel hides surrounding quote marks and can introduce errors by changing the encoding from UTF-8 or otherwise making invisible changes. Instead, we recommend editing CSV files in a text editor like Sublime.

When updating content that is not specific to ongoing branch work, such as fixing a typo or expanding upon existing content, you must make these changes in all branches (all three ongoing branches and the master branch). If an ongoing branch changes something that applies to other simtypes, make that change broadly too. For example, if malaria-ongoing updates the InsetChart report, you must make these changes in all simtype docsets in the malaria-ongoing branch because that report is available for all simtypes.

For more information on the EMOD documentation architecture and content reuse, see EMOD documentation.

Add a new or modify an existing parameter (config, demographics, or campaign)

  1. Add the new parameter description and keystring to the missing_params.rc file in DtkTrunk. This is needed so the schema will compile correctly, but will not add the parameter to the documentation.

  2. File a ticket in the EMOD-docs repository with information about the parameter, including the branch it’s in and the SME to contact for more information. Assign to a member of the content team, who will follow the steps to add it to the paramDB and documentation.

  3. Navigate to the appropriate tab in the paramDB for the type of parameter you want to add or update.

  4. Open an existing parameter or click the Add Parameter button and begin filling in the information:

    Key String

    Use the key string as it appears in the schema, if possible.

    Long Description

    List any dependent parameters and provide links to any related parameters that appear in other input files. For related config and demographics parameters, this means linking to the category it appears in. For related campaign parameters, link to the campaign class. For example, the Blocking_Config parameter description includes a link to the waning effect campaign classes.

    Default/min/max values

    Review the values provided in the schema.

    Simulation Type

    There is no implicit inheritance in the DB anymore. Therefore, if a parameter is provided by the generic sim type and inherited by all others, you must select ALL sim types in this field, not just “generic”.

    JSON Examples

    Accompanying descriptions are optional. The example should include related/dependent parameters in the same file. For example, a configuration parameter should provide an example of this and related parameters in the configuration file. If there are parameters in other input files that are related, do not include them in the JSON example. Link to them in the long description. Campaign parameters should include a brief example without the surrounding intervention, coordinator, and event.

    Object Type

    The data type (float, string, etc.)

    Categories

    For configuration parameters, select up to three categories that this parameter belongs to.

  5. If the parameter is new and has related parameters in the same input file, remember to add the new parameter to the JSON examples in the related parameter entries.

  6. Export the CSVs from the database (lowercase is preferred).

  7. In the emod/csv directory, delete all previously exported CSVs (except those specific to a previous release) and drop the new CSVs into the folder.

  8. Verify that all EMOD docsets build without error.

  9. Commit the new CSVs to GitHub.

  10. The content team member will reassign the original ticket to the person who opened it, and they should do the following:

    1. Run get_params.py to pull the long descriptions into the config or campaign RC files. For example, to generate new config_params.rc and iv_params.rc for version 2.21, you would enter the following:

      get_params.py --config -o config_params.rc --old-naming -v 2.21
      get_params.py --iv -o iv_params.rc --old-naming -v 2.21
      

      Note

      The get_params.py script queries the API for the parameter database. And, without adding the --old-naming flag the script prepends the campaign class name for the _DESC_TEXT name. For example, CalendarEventCoordinator_Node_Property_Restrictions_DESC_TEXT and TriggeredEventCoordinator_Node_Property_Restrictions_DESC_TEXT, etc.

    2. Remove the temporary description and keystring from the missing_params.rc file.

    3. Close the ticket.

Deprecate a parameter (config, demographics, or campaign)

  1. File a ticket in the EMOD-docs repository with information about the parameter, including the branch it’s in and the SME to contact for more information. Assign to a member of the content team, who will follow the steps to remove it from the paramDB and documentation.

  2. Navigate to the appropriate tab in the paramDB for the type of parameter you want to deprecate.

  3. Open the parameter and select the deprecated version from the drop-down menu.

  4. If the parameter has other related parameters, verify that the deprecated parameter is not listed in the JSON examples for any of those parameters.

  5. Export the CSVs from the database (lowercase filenames).

  6. In the emod/csv directory, delete all previously exported CSVs (except those specific to a previous release) and drop the new CSVs into the folder.

  7. In Sublime, go to Find > Find in Files.

  8. Search across all files in the emod folder for the deprecated parameter and delete it. Do the same in the Scenarios directories in the docs-emod-scenarios GitHub repo and verify that the scenarios still run successfully.

  9. Verify that all EMOD docsets build without error.

  10. Commit all changes to GitHub.

  11. Close the ticket.

Add a new campaign class

  1. File a ticket in the EMOD-docs repository with information about the class, including the branch it’s in and the SME to contact for more information. Assign to a member of the content team, who will follow the steps to add it to the paramDB and documentation.

  2. Navigate to the campaign class tab and click Add Class and begin filling in the information:

    Simulation Types

    There is no implicit inheritance in the DB anymore. Therefore, if a parameter is provided by the generic sim type and inherited by all others, you must select ALL sim types in this field, not just “generic”.

    Class/Intervention Type

    Event coordinator, individual intervention, or node intervention.

    Documentation Link

    No longer used.

  3. Add information for ALL parameters that can be used with this class as indicated in Add a new or modify an existing parameter (config, demographics, or campaign).

  4. Export the CSVs from the database (lowercase is preferred).

  5. In the emod/csv directory, delete all previously exported CSVs (except those specific to a previous release) and drop the new CSVs into the folder.

  6. Add a new RST topic for the class to the emod directories:

    • Add the topic to the highest-level simulation type it belongs to (for example, add to generic for a class that is inherited by all simulation types, or add to sti for a class used by STI and HIV simulation types).

    • Include a full description of the class.

    • Include a full JSON example (with event, event coordinator, and intervention).

    • Include the csv-table directive with all parameters belonging to the class.

  7. Add the new class topic to the TOC.

  8. In all simulation types that inherit the classes, add an empty topic of the same name with an “include” directive pointing to the new topic. Add this to the TOCs.

  9. Verify that all EMOD docsets build without error.

  10. Commit all changes to GitHub.

  11. The content team member will reassign the original ticket to the person who opened it, and they should do the following:

    1. Run get_params.py to pull the long descriptions into the config or campaign RC files. For example, to generate new config_params.rc and iv_params.rc for version 2.21, you would enter the following:

      get_params.py --config -o config_params.rc --old-naming -v 2.21
      get_params.py --iv -o iv_params.rc --old-naming -v 2.21
      

      Note

      The get_params.py script queries the API for the parameter database. And, without adding the --old-naming flag the script prepends the campaign class name for the _DESC_TEXT name. For example, CalendarEventCoordinator_Node_Property_Restrictions_DESC_TEXT and TriggeredEventCoordinator_Node_Property_Restrictions_DESC_TEXT, etc.

    2. Remove the temporary description and keystring from the missing_params.rc file.

    3. Close the ticket.

Deprecate a campaign class

  1. File a ticket in the EMOD-docs repository with information about the class, including the branch it’s in and the SME to contact for more information. Assign to a member of the content team, who will follow the steps to add it to the paramDB and documentation.

  2. Open the class and select the deprecated version from the drop-down menu.

  3. Export the CSVs from the database (lowercase filenames).

  4. In the emod/csv directory, delete all previously exported CSVs (except those specific to a previous release) and drop the new CSVs into the folder. This is particularly important for class deprecations.

  5. In Sublime, go to Find > Find in Files.

  6. Search across all files in the emod folder for the deprecated class and delete it. Do the same in the Scenarios directories in the docs-emod-scenarios GitHub repo. For scenarios, contact the SME to see what class to use instead and verify that the scenarios run.

  7. Delete the JSON file that provides the full example for the class.

  8. Verify that all EMOD docsets build without error.

  9. Commit all changes to GitHub.

  10. Close the ticket.