vis_tools.VisSet module

VisSet.py

This class generates the visset.json file that is the input for the Vis-Tools HTML client component. It reads defaults from a file called “defaultvisset.json” and extends itself with that. The caller fills in some vital information and then can overrides defaults the object with various method calls, then finally emits the visset.json file.

Note

This file will eternally appear to have PEP errors in it because it makes reference to members that are not initially created in the constructor. Those members are acquired when then defaultvisset.json is read in and used to extend self. So these PEP errors are not actually errors.

Minimal usage:

vis_set = VisSet("MySim")
vis_set.set_target_client("Geospatial")
vis_set.set_node_vis_type("Points")
vis_set.set_products_directory("my_sim/output/")
vis_set.set_demographics("my_sim/demographics.json")
vis_set.add_spatial_reports(SpatialReports("my_sim/output/"))
vis_set.validate()
vis_set.write()
class vis_tools.VisSet.VisSet(name='Untitled', verbose=True)

Bases: object

Class that generates visset.json files.

This class provides a friendly way to generate a customized visset.json for Vis-Tools various HTML clients.

Rather than incorporating its defaults directly in code, it instead reads its defaults from defaultvisset.json. This important, because that same file is used by the HTML clients, so that the defaults for both remain in sync.

VisSet extends itself from defaultvisset.json, which is to say, all of the fields from defaultvisset.json become part of the VisSet Python object itself.

There are methods to allow easy access to some aspects of the VisSet, such as adding bindings, showing/hiding sections, and adding various types of visual layers. But the visset data structure is a public part of this object and it is allowable for a preprocessing script to directly poke into it whatever customizations are required. E.g.:

vis_set.options["defaultBaseLayer"] = "ESRI National Geographic"
vis_set.options["insetCharts"]["defaultChannelName"] = "Prevalence"
Public members:

name (str): The name (given in the constructor).

options (obj): The options section based on the targetClient

all keys found in defaultvisset.json

k_default_visset = 'defaultvisset.json'
k_visset_file_name = 'visset.json'
k_infected_migrations_file_name = 'InfectedMigrations.czml'
k_vector_migrations_file_name = 'VectorMigrations.czml'
k_network_layer_file_name = 'Network.czml'
k_geospatial = 'Geospatial'
k_errno_permission_denied = 13
k_nodes_as_points = 'Points'
k_nodes_as_shapes = 'Shapes'
k_node_shape_square = 'Square'
k_marker_shape_triangle = 'Triangle'
k_marker_shape_cross = 'Cross'
k_marker_placement_top = 'Top'
k_marker_placement_left = 'Left'
k_marker_placement_bottom = 'Bottom'
k_marker_placement_right = 'Right'
k_default_marker_duration_days = 2
set_sim_id(sim_id)

Set a sim_id to be associated with this VisSet.

Returns

None.

Parameters

sim_id (str) – Simulation ID as a GUID.

set_asset_map_path(asset_map_path)

Set the path to the optional asset map.

Returns

None.

Parameters
  • asset_map_path (str) – Path to a VtAssetMap.json file containing the

  • between local asset paths and COMPS asset manager URLs. If (mapping) –

  • map is present, then the output visset.json will contain both (this) –

  • paths and asset manager URLs. (local) –

set_verbose(verbose)

Set whether the class emits verbose messages.

Returns

None.

Parameters

verbose (bool) – True for verbose output, False for relative silence.

get_output_directory()

Getter for the output directory, where visset.json will be written. THIS METHOD IS DEPRECATED. Use get_products_dir instead.

Returns

Currently set output directory or None.

Return type

str

Parameters

None.

get_products_directory()

Getter for the products directory, where the visset.json and other artifact files are written.

This function replaces get_output_directory, which had a name that could lead to confusion with the simulation’s output/ directory.

Returns

Currently set products directory or None.

Return type

str

Parameters

None.

get_config()

Getter for the config object.

Returns

Currently set Config object or None.

Return type

obj

Parameters

None.

get_demographics()

Getter for the demographics object.

Returns

Currently set Demographics object or None.

Return type

obj

Parameters

None.

get_event_recorder()

Getter for the event recorder object.

Returns

Currently set CSVReport object for event recorder, or None.

Return type

obj

Parameters

None.

set_target_client(client_name)

Setter for the selected target client.

This method sets the target client for the output visset. This causes the public member options to get overwritten with the defaults for the new client name. The allowable client names are provided as class constants.

Returns

None.

Parameters

client_name (str) – “Geospatial” is the only target right now.

set_node_vis_type(node_vis_type)

Setter for the node visualization type.

For the Geospatial visualization client, there are two fundamental ways nodes may be represented: Points and Shapes. The former are 2D dots, measured in pixels and always perpendicular to the eye vector, which can be sized and styled variously. The latter uses 3D boxes, measured in meters, which can be styled as well but differently from Points. This setter is where the caller chooses which representation they want for their nodes. Each node visualization type has its own suite of settings.

The allowable node_vis_type values are provided as class constants.

For:

Geospatial visualization.

Returns

None.

Parameters

node_vis_type (str) – “Points” or “Shapes”. See constants section.

set_output_directory(output_directory)

Setter for the output directory. THIS METHOD IS DEPRECATED. Use set_products_directory instead.”

This setter must be called before write().

Returns

None.

Parameters
  • output_directory (str) – the directory in which to write the output

  • file. (visset.json) –

set_products_directory(products_directory)

Setter for products dir, where preprocessing products are written.

This setter must be called before write() or other methods that write preprocessing artifacts. This method replaces the now-deprecated set_output_directory method, which was replaced because of possible confusion with the simulation output/ directory.

Returns

None.

Parameters
  • products_directory (str) – the directory into which to write the

  • visset.json and other preprocessing artifact files. (output) –

set_demographics(demographics_file_path)

Setter for the demographics file path.

This setter must be called before write(). The Geospatial client requires node locations for visualization. An internal Demographics object is created using the given path. If it is useful to access that object directly it can be obtained using the corresponding getter method. Note that if you have multiple demographics files (i.e. overlays) you can call set_demographics_files() instead.

For:

Geospatial visualization.

Returns

None.

Parameters

demographics_file_path (str) – Path to demographics file.

Raises

I/O and JSON exceptions.

set_demographics_files(demographics_file_list)

Setter for demographics from multiple overlay files.

This setter must be called before write(). The Geospatial client requires node locations for visualization. An internal Demographics object is created using the given path. If it is useful to access that object directly it can be obtained using the corresponding getter method. Note that if you have just a single demographics file you can call set_demographics() instead.

For:

Geospatial visualization.

Returns

None.

Parameters

demographics_file_list (list) – Paths to demographics files.

Raises

I/O and JSON exceptions.

set_config(config_file_path)

Setter for config file path.

The VisSet needs to know how many timesteps are in the simulation. This can be obtained from any spatial binary, or from the config. This method allows you to specify the path to the config.json, and the VisSet will read the Simulation_Duration from there. Use this method if your simulation does not have any spatial binaries, and you just want to look at the nodes.

Note

If you call add_spatial_reports() and there is at least one spatial binary, the VisSet will obtain the timestep count from that, and you do not need to call set_config at all.

Returns

None.

Parameters

config_file_path (str) – Path to config.json file.

Raises

I/O and JSON exceptions.

set_event_recorder(event_recorder_file_path)

Setter for event recorder file path for marker layers.

This setter must be called before any calls to add_marker_layer(). An internal CSVReport object will be created with the data at the given path.

For:

Geospatial visualization.

Returns

None.

Parameters

event_recorder_file_path (str) – Path to an ReportEventRecorder.csv.

Raises

I/O and csv exceptions.

set_inset_chart(inset_chart_file_path)

Setter for inset chart data source file.

The Geospatial Vis-Tools client has an Inset Panel which shows inset chart data. Typically that data is sourced from an InsetChart.json file, but any file with the same format is allowable, such as ReportMalariaFiltered.json. This method allows the user to specify the source file for Insets.

Note

If this method is not called with some json file path, the Inset Panel will not appear at all in the Geospatial client.

Returns

None.

Parameters
  • inset_chart_file_path (str) – File path to InsetChart.json or other

  • file. (compatible) –

set_custom_base_layer(base_layer_path, friendlyName=None)

Setter for custom base layer source file.

The Geospatial Vis-Tools client allows the use of a single-tile globe image as its base layer. Such an image needs to be a high-resolution JPEG or PNG image that is an equirectangular projection.

Returns

None.

Parameters

base_layer_path (str) – File path to a JPEG or PNG base layer image.

set_start_date(yyyy_mm_dd)

Setter for simulation start date.

If this is not given, the start date (that shows in the timeline in Vis-Tools visualizations) will be back-calculated from the number of timesteps such that the simulation ends on the date on which the preprocessing script was run. This setter allows you to set an arbitrary start date, and the timeline will start there instead.

Returns

None.

Parameters

yyyy_mm_dd (str) – a date in the form “yyyy-dd-mm”, e.g. “1970-01-01”

add_node_field(field_name)

Allows the caller to pull in additional fields from NodeAttributes.

The output visset.json contains a summarized version of the nodes that are read from the demographics file. By default that summary includes latitude, longitude, and altitude if present. This method allows the caller to pull in any additional desired fields from NodeAttributes into the node summaries in the output visset.json. E.g.:

vis_set.add_node_field("BirthRate")

Note

Automatically calls update_nodes(). Caller does not need to do that.

Returns

None.

Parameters

field_name (str) – The NodeAttributes field name to pull in.

add_spatial_reports(spatial_reports)

Adds all the spatial reports in the given SpatialReports object.

This method adds as sources all the spatial report files referenced in the given SpatialReports object. Note that the current Geospatial client HTML loads all the spatial binaries it knows about into memory, so it may make sense to edit down the set of reports that are included. That can be done (subsequent to calling this method) using exclude_spatial_channels().

Note

This method also, as a side effect, determines the number of timesteps in the simulation by inspecting the first spatial binary. As a result, if spatial binaries are provided, a config.json need not be added to the VisSet object.

Returns

None.

Parameters

spatial_reports (obj) – the SpatialReports object to add.

add_marker_layer(event_name, marker=None, color=None, placement=None, duration=None, show=True)

Adds a marker animation layer to the output visset.

This method generates a CZML animation layer that provides visual indication of event recorder events.

Note

Call set_products_directory, set_demographics, set_event_recorder, and (add_spatial_channels or set_config) before calling this method.

For:

Geospatial visualization.

Returns

None. Creates a CZML file in the VisSet’s products directory.

Parameters
  • event_name (str) – The name of the event for which a marker layer is

  • It must exactly match the event name in the event (desired.) –

  • E.g. "Received_Treatment". (recorder.) –

  • marker (str) – The name of the marker image to use. “Triangle” or

  • are allowable ("Cross") –

  • color (str) – An HTML/CSS or SVG named color for the marker. E.g.

  • or "#ccff00" ("red") –

  • placement (str) – The location, with respect to the node lat/log

  • for the marker. "Top", "Left", "Bottom", or "Right" are (center,) –

  • allowable (see constants at top of class) –

  • duration (int) – The duration, in timesteps for which the marker will

  • visible after the event occurs. E.g. 2 will cause the (remain) –

  • to appear above the node on the timestep of the event, (marker) –

  • remain there for two timesteps before disappearing. (and) –

  • show (bool) – Whether this layer will be visible by default when the

  • client loads this visset. If show is False, the layer (Geospatial) –

  • is still generated and loaded by the client, but the layer (CZML) –

  • have its checkbox unchecked (will) –

  • the client when the page is first loaded. (in) –

add_infected_human_migrations(human_migration_file_path)

Adds an infected human migrations layer to the output visset.

Note

Call set_products_directory, set_demographics, set_event_recorder, and (add_spatial_channels or set_config) before calling this method.

For:

Geospatial visualization.

Returns

None. Creates a CZML file in the VisSet’s products directory.

Parameters
  • human_migration_file_path (str) – The file path to the simulation’s

  • file. (ReportHumanMigrationTracking.csv) –

To do:
  • Add arg for comet head start color

  • Add arg for comet head end color

  • Add arg for comet tail start color

  • Add arg for comet tail end color

  • Add arg for tail duration in timesteps

add_vector_migrations(vector_migration_path, migration_duration_timesteps=2, dot_color='#ba7fb7', dot_size_pixels=10, path_color='#ffffff', path_thickness_pixels=2.5, path_trail_time_factor=0.75)

Adds a vector cohort migration layer to the output visset.

Note

Call set_products_directory, set_demographics, and either set_config or add_spatial_reports before calling this method.

For:

Geospatial visualization.

Returns

None. Creates a CZML file in the VisSet’s products directory.

Parameters
  • vector_migration_path (str) – The path to the

  • file. (ReportVectorMigration.csv) –

  • migration_duration_timesteps (int) – Number of timesteps that the

  • animation spans. (migration) –

  • dot_color (string) – CSS #rrggbb color for migration comet dot.

  • dot_size_pixels (int) – Size in pixels of comet dot.

  • path_color (string) – CSS #rrggbb color for migration comet tail.

  • path_thickness_pixels (float) – Thickness in pixels of comet tail.

  • path_trail_time_factor (float) – Length of trail as a multiple of

  • migration_duration. E.g. if this is 1.0, the trail length will (the) –

  • the full distance from source node to the destination node. If (be) –

  • the trail length will be half the distance between the nodes. (0.5,) –

add_simplified_vector_migrations(vector_migration_path, migration_duration_timesteps=2, arrow_color='#ffffff', arrow_thickness_pixels=2)

Adds a simplified vector cohort migration layer to the output visset.

Note

Call set_products_directory, set_demographics, and either set_config or add_spatial_reports before calling this method.

For:

Geospatial visualization.

Returns

None. Creates a CZML file in the VisSet’s products directory.

Parameters
  • vector_migration_path (str) – The path to the

  • file. (ReportVectorMigration.csv) –

  • migration_duration_timesteps (int) – Number of timesteps that the

  • animation spans. (migration) –

  • arrow_color (string) – CSS #rrggbb color for migration arrow.

  • arrow_thickness_pixels (float) – Thickness in pixels of arrow.

add_binding(vis_section, sink_name, source_name, func)

Add a binding between a data source and a visualization sink.

This method lets you add a binding between a visualization sink (e.g. point node color) and a data source (e.g. a spatial report) through a binding function (e.g. “scale(3, 20)”). Such bindings will be preset in the client page when it is first presented from the visset.

Note

The available sinks vary depending on target client, and within a target client, on other factors. For example, there is one set of sinks for “Points” node representation, and a different set of sinks for “Shapes” node representation. You can see the list of sinks by inspecting the defaultvisset.json file.

Note

The correctness of the functions provided in the func argument is not validated here. If the caller makes an error in the syntax of the function string, the errant string will still end up in the output visset.json. When the client loads the visset, the binding will throw an exception and the client will show an error message.

Returns

True if the binding was set, or False if the sink could not be found.

Return type

bool

Parameters
  • vis_section (str) – The visualization section of the visset.json file. For the Geospatial client, valid values for vis_section are “nodeVis” or “heatmapVis”. Other clients will have other visualization sections. See defaultvisset.json.

  • sink_name (str) – The name of the sink to which to bind. E.g. “shapeExtrusion”.

  • source_name (str) – The name of the data source to which to bind. E.g. “SpatialReport_Prevalence” or “BirthRate”. Can be None.

  • func (str) –

    The function to use on the binding or None. The details of function syntax are documented elsewhere, but in general this may be “none()”, some built-in function e.g. “scale(3, 20)”, or a custom Javascript function body, e.g.:

    "{ return 1.0 - binding.value; }"
    

static opacity_one(weight, norm_weight)
add_weighted_network_layer(network, gradient_spec, layer_name='Network', layer_file_name='Network', opacity_func=None)

Adds a weighted network visualization layer to the output visset.

This method generates a CZML layer that provides a visual representation of a weighted network between nodes.

Note

Call set_products_directory and set_demographics before calling this method.

For:

Geospatial visualization.

Returns

None. Creates a CZML file in the VisSet’s products directory.

Parameters
  • network (array) –

    array of objects:

    {
        from: <from-node-id>,
        to: <to-node-id>,
        weight: <float-weight>
    }
    

  • gradient_spec (str) – gradient spec for a gradient with which to

  • the network lines. (color) –

  • layer_name (str) – Layer name (to be displayed in the client UI).l

  • layer_file_name (str) – The layer base name for the output file. For

  • "Network" results in an output file called "Network.czml". (example,) –

  • opacity_func (function) – function(weight, norm_weight) that returns

  • desired opacity in range [0,1] (the) –

show_layer(vis_section)

Sets a layer to be shown by default in the output visset.json.

This method lets the caller set that the particular visualization section is shown by default when the visset is loaded by the client.

Returns

None.

Parameters
  • vis_section (str) – The visualization section of the visset.json

  • For the Geospatial client, valid values for vis_section (file.) –

  • "nodeVis" or "heatmapVis". Other clients will have other (are) –

  • sections. See defaultvisset.json. (visualization) –

hide_layer(vis_section)

Sets a layer to be hidden by default in the output visset.json.

This method lets the caller set that the particular visualization section is hidden by default when the visset is loaded by the client. Note that all the data and bindings for that visualization section are still created and emitted to the visset…only their default visibility is affected.

Returns

None.

Parameters
  • vis_section (str) – The visualization section of the visset.json

  • For the Geospatial client, valid values for vis_section (file.) –

  • "nodeVis" or "heatmapVis". Other clients will have other (are) –

  • sections. See defaultvisset.json. (visualization) –

exclude_spatial_channels(spatial_channel_names)

Exclude named channels from the list of binding sources.

Allows caller to exclude a set of spatial channels from showing up as sources for visualization bindings. This would primarily be done to reduce the memory pressure (or load time) of the client page, since all the source spatial binaries have to be loaded into memory on the client side. All this really does is set show=False in the link, which excludes the source on the client side.

Note

Call add_spatial_reports() before calling this method.

Returns

None.

Parameters

spatial_channel_names (list) –

a list or array of spatial channel names. If only one channel is to be excluded, enclose it in parens or [] so that it is iterable. E.g.:

exclude_spatial_channels(["SpatialReport_Adult_Vectors"])

include_spatial_channels(spatial_channel_names)

Include (only) named channels in the list of binding sources.

Allows caller to include a subset of the available spatial channels as the list of possible sources for visualization bindings. This would primarily be done to reduce the memory pressure (or load time) of the client page, since all the source spatial binaries have to be loaded into memory on the client side. All this really does is set the specified channels to show=True and all the rest to show=False.

Note

Call add_spatial_reports() before calling this method.

Returns

None.

Parameters

spatial_channel_names (list) –

a list or array of spatial channel names. If only one channel is to be excluded, enclose it in parens or [] so that it is iterable e. E.g.:

include_spatial_channels(["SpatialReport_Adult_Vectors"])

exclude_nodes_from_stats(node_id_array)

Excludes a set of nodes from node min/max statistics.

This method lets the caller specify a list of node ids that should be excluded from the min/max calculations data channels. This includes both static sources (e.g. NodeAttributes.InitialPopulation) and spatial reports (e.g. SpatialReport_Population). This is typically used to drop an outlier from the data set to avoid visually skewing the output visualization. For example, one might exclude the “elsewhere” node, which typically has extraordinary population, prevalence, etc. in a simulation that incorporates external migration.

Note

The node itself is not eliminated - it will still be visually represented, but it will have a value that may lie outside the min/ max range calculated for a given source. The client will clamp the value at visualization time.

Note

This method calls update_nodes…the caller need not do that.

Returns

None.

Parameters

node_id_array (list) –

a list or array of node id’s to be excluded. If only a single node is to be excluded, enclose it in parens or [] so that it is iterable, e.g.:

exclude_nodes_from_stats([10001])

or just call exclude_node_from_stats()

exclude_node_from_stats(node_id)

Exclude a node from node min/max statistics.

This method is exactly exclude_nodes_from_stats but for a single node id. See that method for details.

Returns

None.

Parameters

node_id (int) – Node id to exclude from stats.

update_nodes()

Updates the local summarized nodes from the Demographics source.

This method forces the VisSet object to refresh its summarized copy of the node data that it will later include in the written visset.json. Use this method if you change the data in the Demographics object and want those changes reflected in the output visset.json.

Returns

None.

Parameters

None.

update_ranges()

Updates the local min/max values for extra node fields.

This method updates the local min/max values that are calculated and later emitted into the visset.json file. Use this method if you change the values in the Demographics object.

Returns

None.

Parameters

None.

validate()

Validate the settings in the visset to ensure their validity.

This function validates the visset to increase the chances of it working correctly when brought into the client.

Returns

None.

Parameters

None.

To do:
  • Validate targetClient

  • Validate nodeVis.visType

  • Validate that self.nodes is populated

  • Validate that self.timestepCount and startDate are populated

  • Try to guess whether the list of spatial binaries is too big to be loaded on the client side

  • Validate binding sources have show=true

  • Validate binding functions to the extent possible

  • Validate that the inset chart file is a JSON and contains the expected keys

write(sim_url_root=None)

Write the customized VisSet object to a visset.json file in the products directory.

This method is called after the caller has made all their additions and modifications to the default visset to write a visset.json file to the products directory.

Returns

None. Writes a “visset.json” file to the VisSet’s products directory.

Parameters
  • sim_url_root (str) – Optional. Url root substitution if desired. If

  • provided, URLs in the output visset will begin with "./". (not) –

Raises

I/O and JSON exceptions.

write_as(file_name, sim_url_root=None)

Write the customized VisSet object to a specified JSON file.

This method is called after the caller has made all their additions and modifications to the default visset to write a JSON file with the given name to the products directory.

Returns

None. Writes a JSON file to the VisSet’s products directory.

Parameters
  • file_name (str) – The file name to use for the output visset JSON

  • file.

  • sim_url_root (str) – Optional. Url root substitution if desired. If

  • provided, URLs in the output visset will begin with "./". (not) –

Raises

I/O and JSON exceptions.

static update_urls(visset, asset_map_or_url_root)

Integrate COMPS Asset Manager URLs or update url roots.

This function does two different things, but they are encapsulated into this one function because this function is the only spot in VisSet that knows where all the URLs are in visset[links].

If passed an asset map, it adds “url_asset” entries next to “url” entries to allow the visset to be used in a hosted environment.

If passed a string, the “url” entries are updated to replace “./” with the given root. This makes the local paths suitable for use with a locally-hosted Vis-Tools.

Returns

None.

Parameters
  • visset (obj) – A visset read with json.load or a VisSet __dict__.

  • asset_map (obj|str) – Either an asset map read with json.load, or

  • string to be the new root for urls. (a) –