vis_tools.VisSet module

VisSet.py

This class generates the visset.json file that is the input for the Vis-Tools HTML client component. It reads defaults from a file called “defaultvisset.json” and extends itself with that. The caller fills in some vital information and then can overrides defaults the object with various method calls, then finally emits the visset.json file.

Note

This file will eternally appear to have PEP errors in it because it makes reference to members that are not initially created in the constructor. Those members are acquired when then defaultvisset.json is read in and used to extend self. So these PEP errors are not actually errors.

Minimal usage:

vis_set = VisSet("MySim")
vis_set.set_target_client("Geospatial")
vis_set.set_node_vis_type("Points")
vis_set.set_products_directory("my_sim/output/")
vis_set.set_demographics("my_sim/demographics.json")
vis_set.add_spatial_reports(SpatialReports("my_sim/output/"))
vis_set.validate()
vis_set.write()

class vis_tools.VisSet.VisSet(name='Untitled', verbose=True)

Bases: object

Class that generates visset.json files.

This class provides a friendly way to generate a customized visset.json for Vis-Tools various HTML clients.

Rather than incorporating its defaults directly in code, it instead reads its defaults from defaultvisset.json. This important, because that same file is used by the HTML clients, so that the defaults for both remain in sync.

VisSet extends itself from defaultvisset.json, which is to say, all of the fields from defaultvisset.json become part of the VisSet Python object itself.

There are methods to allow easy access to some aspects of the VisSet, such as adding bindings, showing/hiding sections, and adding various types of visual layers. But the visset data structure is a public part of this object and it is allowable for a preprocessing script to directly poke into it whatever customizations are required. E.g.:

vis_set.options["defaultBaseLayer"] = "ESRI National Geographic"
vis_set.options["insetCharts"]["defaultChannelName"] = "Prevalence"

Public members:

name (str): The name (given in the constructor).

options (obj): The options section based on the targetClient

all keys found in defaultvisset.json

k_default_visset = 'defaultvisset.json'

k_visset_file_name = 'visset.json'

k_infected_migrations_file_name = 'InfectedMigrations.czml'

k_vector_migrations_file_name = 'VectorMigrations.czml'

k_network_layer_file_name = 'Network.czml'

k_geospatial = 'Geospatial'

k_errno_permission_denied = 13

k_nodes_as_points = 'Points'

k_nodes_as_shapes = 'Shapes'

k_node_shape_square = 'Square'

k_marker_shape_triangle = 'Triangle'

k_marker_shape_cross = 'Cross'

k_marker_placement_top = 'Top'

k_marker_placement_left = 'Left'

k_marker_placement_bottom = 'Bottom'

k_marker_placement_right = 'Right'

k_default_marker_duration_days = 2

set_sim_id(sim_id)

Set a sim_id to be associated with this VisSet.

Returns:

None.

Parameters:

sim_id (str) – Simulation ID as a GUID.

set_asset_map_path(asset_map_path)

Set the path to the optional asset map.

Returns:

None.

Parameters:

• asset_map_path (str) – Path to a VtAssetMap.json file containing the

• If (mapping between local asset paths and COMPS asset manager URLs.) –

• present (this map is) –

• both (then the output visset.json will contain) –

• URLs. (local paths and asset manager) –

set_verbose(verbose)

Set whether the class emits verbose messages.

Returns:

None.

Parameters:

verbose (bool) – True for verbose output, False for relative silence.

get_output_directory()

Getter for the output directory, where visset.json will be written. THIS METHOD IS DEPRECATED. Use get_products_dir instead.

Returns:

Currently set output directory or None.

Return type:

str

Parameters:

None. –

get_products_directory()

Getter for the products directory, where the visset.json and other artifact files are written.

This function replaces get_output_directory, which had a name that could lead to confusion with the simulation’s output/ directory.

Returns:

Currently set products directory or None.

Return type:

str

Parameters:

None. –

get_config()

Getter for the config object.

Returns:

Currently set Config object or None.

Return type:

obj

Parameters:

None. –

get_demographics()

Getter for the demographics object.

Returns:

Currently set Demographics object or None.

Return type:

obj

Parameters:

None. –

get_event_recorder()

Getter for the event recorder object.

Returns:

Currently set CSVReport object for event recorder, or None.

Return type:

obj

Parameters:

None. –

set_target_client(client_name)

Setter for the selected target client.

This method sets the target client for the output visset. This causes the public member options to get overwritten with the defaults for the new client name. The allowable client names are provided as class constants.

Returns:

None.

Parameters:

client_name (str) – “Geospatial” is the only target right now.

set_node_vis_type(node_vis_type)

Setter for the node visualization type.

For the Geospatial visualization client, there are two fundamental ways nodes may be represented: Points and Shapes. The former are 2D dots, measured in pixels and always perpendicular to the eye vector, which can be sized and styled variously. The latter uses 3D boxes, measured in meters, which can be styled as well but differently from Points. This setter is where the caller chooses which representation they want for their nodes. Each node visualization type has its own suite of settings.

The allowable node_vis_type values are provided as class constants.

For:

Geospatial visualization.

Returns:

None.

Parameters:

node_vis_type (str) – “Points” or “Shapes”. See constants section.

set_output_directory(output_directory)

Setter for the output directory. THIS METHOD IS DEPRECATED. Use set_products_directory instead.”

This setter must be called before write().

Returns:

None.

Parameters:

• output_directory (str) – the directory in which to write the output

• file. (visset.json) –

set_products_directory(products_directory)

Setter for products dir, where preprocessing products are written.

This setter must be called before write() or other methods that write preprocessing artifacts. This method replaces the now-deprecated set_output_directory method, which was replaced because of possible confusion with the simulation output/ directory.

Returns:

None.

Parameters:

• products_directory (str) – the directory into which to write the

• files. (output visset.json and other preprocessing artifact) –

set_demographics(demographics_file_path)

Setter for the demographics file path.

This setter must be called before write(). The Geospatial client requires node locations for visualization. An internal Demographics object is created using the given path. If it is useful to access that object directly it can be obtained using the corresponding getter method. Note that if you have multiple demographics files (i.e. overlays) you can call set_demographics_files() instead.

For:

Geospatial visualization.

Returns:

None.

Parameters:

demographics_file_path (str) – Path to demographics file.

Raises:

I/O and JSON exceptions. –

set_demographics_files(demographics_file_list)

Setter for demographics from multiple overlay files.

This setter must be called before write(). The Geospatial client requires node locations for visualization. An internal Demographics object is created using the given path. If it is useful to access that object directly it can be obtained using the corresponding getter method. Note that if you have just a single demographics file you can call set_demographics() instead.

For:

Geospatial visualization.

Returns:

None.

Parameters:

demographics_file_list (list) – Paths to demographics files.

Raises:

I/O and JSON exceptions. –

set_config(config_file_path)

Setter for config file path.

The VisSet needs to know how many timesteps are in the simulation. This can be obtained from any spatial binary, or from the config. This method allows you to specify the path to the config.json, and the VisSet will read the Simulation_Duration from there. Use this method if your simulation does not have any spatial binaries, and you just want to look at the nodes.

Note

If you call add_spatial_reports() and there is at least one spatial binary, the VisSet will obtain the timestep count from that, and you do not need to call set_config at all.

Returns:

None.

Parameters:

config_file_path (str) – Path to config.json file.

Raises:

I/O and JSON exceptions. –

set_event_recorder(event_recorder_file_path)

Setter for event recorder file path for marker layers.

This setter must be called before any calls to add_marker_layer(). An internal CSVReport object will be created with the data at the given path.

For:

Geospatial visualization.

Returns:

None.

Parameters:

event_recorder_file_path (str) – Path to an ReportEventRecorder.csv.

Raises:

I/O and csv exceptions. –

set_inset_chart(inset_chart_file_path)

Setter for inset chart data source file.

The Geospatial Vis-Tools client has an Inset Panel which shows inset chart data. Typically that data is sourced from an InsetChart.json file, but any file with the same format is allowable, such as ReportMalariaFiltered.json. This method allows the user to specify the source file for Insets.

Note

If this method is not called with some json file path, the Inset Panel will not appear at all in the Geospatial client.

Returns:

None.

Parameters:

• inset_chart_file_path (str) – File path to InsetChart.json or other

• file. (compatible) –

set_custom_base_layer(base_layer_path, friendlyName=None)

Setter for custom base layer source file.

The Geospatial Vis-Tools client allows the use of a single-tile globe image as its base layer. Such an image needs to be a high-resolution JPEG or PNG image that is an equirectangular projection.

Returns:

None.

Parameters:

base_layer_path (str) – File path to a JPEG or PNG base layer image.

set_start_date(yyyy_mm_dd)

Setter for simulation start date.

If this is not given, the start date (that shows in the timeline in Vis-Tools visualizations) will be back-calculated from the number of timesteps such that the simulation ends on the date on which the preprocessing script was run. This setter allows you to set an arbitrary start date, and the timeline will start there instead.

Returns:

None.

Parameters:

yyyy_mm_dd (str) – a date in the form “yyyy-dd-mm”, e.g. “1970-01-01”

add_node_field(field_name)

Allows the caller to pull in additional fields from NodeAttributes.

The output visset.json contains a summarized version of the nodes that are read from the demographics file. By default that summary includes latitude, longitude, and altitude if present. This method allows the caller to pull in any additional desired fields from NodeAttributes into the node summaries in the output visset.json. E.g.:

vis_set.add_node_field("BirthRate")

Note

Automatically calls update_nodes(). Caller does not need to do that.

Returns:

None.

Parameters:

field_name (str) – The NodeAttributes field name to pull in.

add_spatial_reports(spatial_reports)

Adds all the spatial reports in the given SpatialReports object.

This method adds as sources all the spatial report files referenced in the given SpatialReports object. Note that the current Geospatial client HTML loads all the spatial binaries it knows about into memory, so it may make sense to edit down the set of reports that are included. That can be done (subsequent to calling this method) using exclude_spatial_channels().

Note

This method also, as a side effect, determines the number of timesteps in the simulation by inspecting the first spatial binary. As a result, if spatial binaries are provided, a config.json need not be added to the VisSet object.

Returns:

None.

Parameters:

spatial_reports (obj) – the SpatialReports object to add.

add_marker_layer(event_name, marker=None, color=None, placement=None, duration=None, show=True)

Adds a marker animation layer to the output visset.

This method generates a CZML animation layer that provides visual indication of event recorder events.

Note

Call set_products_directory, set_demographics, set_event_recorder, and (add_spatial_channels or set_config) before calling this method.

For:

Geospatial visualization.

Returns:

None. Creates a CZML file in the VisSet’s products directory.

Parameters:

• event_name (str) – The name of the event for which a marker layer is desired. It must exactly match the event name in the event recorder. E.g. “Received_Treatment”.

• marker (str) – The name of the marker image to use. “Triangle” or “Cross” are allowable (see constants at top of class).

• color (str) – An HTML/CSS or SVG named color for the marker. E.g. “red” or “#ccff00”

• placement (str) – The location, with respect to the node lat/log center, for the marker. “Top”, “Left”, “Bottom”, or “Right” are allowable (see constants at top of class).

• duration (int) – The duration, in timesteps for which the marker will remain visible after the event occurs. E.g. 2 will cause the marker to appear above the node on the timestep of the event, and remain there for two timesteps before disappearing.

• show (bool) – Whether this layer will be visible by default when the Geospatial client loads this visset. If show is False, the layer CZML is still generated and loaded by the client, but the layer will have its checkbox unchecked (and therefore not be visible) in the client when the page is first loaded.

add_infected_human_migrations(human_migration_file_path)

Adds an infected human migrations layer to the output visset.

Note

Call set_products_directory, set_demographics, set_event_recorder, and (add_spatial_channels or set_config) before calling this method.

For:

Geospatial visualization.

Returns:

None. Creates a CZML file in the VisSet’s products directory.

Parameters:

• human_migration_file_path (str) – The file path to the simulation’s

• file. (ReportHumanMigrationTracking.csv) –

To do:

• Add arg for comet head start color

• Add arg for comet head end color

• Add arg for comet tail start color

• Add arg for comet tail end color

• Add arg for tail duration in timesteps

add_vector_migrations(vector_migration_path, migration_duration_timesteps=2, dot_color='#ba7fb7', dot_size_pixels=10, path_color='#ffffff', path_thickness_pixels=2.5, path_trail_time_factor=0.75)

Adds a vector cohort migration layer to the output visset.

Note

Call set_products_directory, set_demographics, and either set_config or add_spatial_reports before calling this method.

For:

Geospatial visualization.

Returns:

None. Creates a CZML file in the VisSet’s products directory.

Parameters:

• vector_migration_path (str) – The path to the

• file. (ReportVectorMigration.csv) –

• migration_duration_timesteps (int) – Number of timesteps that the

• spans. (migration animation) –

• dot_color (string) – CSS #rrggbb color for migration comet dot.

• dot_size_pixels (int) – Size in pixels of comet dot.

• path_color (string) – CSS #rrggbb color for migration comet tail.

• path_thickness_pixels (float) – Thickness in pixels of comet tail.

• path_trail_time_factor (float) – Length of trail as a multiple of

• 1.0 (the migration_duration. E.g. if this is) –

• will (the trail length) –

• If (be the full distance from source node to the destination node.) –

• 0.5 –

• nodes. (the trail length will be half the distance between the) –

add_simplified_vector_migrations(vector_migration_path, migration_duration_timesteps=2, arrow_color='#ffffff', arrow_thickness_pixels=2)

Adds a simplified vector cohort migration layer to the output visset.

Note

Call set_products_directory, set_demographics, and either set_config or add_spatial_reports before calling this method.

For:

Geospatial visualization.

Returns:

None. Creates a CZML file in the VisSet’s products directory.

Parameters:

• vector_migration_path (str) – The path to the

• file. (ReportVectorMigration.csv) –

• migration_duration_timesteps (int) – Number of timesteps that the

• spans. (migration animation) –

• arrow_color (string) – CSS #rrggbb color for migration arrow.

• arrow_thickness_pixels (float) – Thickness in pixels of arrow.

add_binding(vis_section, sink_name, source_name, func)

Add a binding between a data source and a visualization sink.

This method lets you add a binding between a visualization sink (e.g. point node color) and a data source (e.g. a spatial report) through a binding function (e.g. “scale(3, 20)”). Such bindings will be preset in the client page when it is first presented from the visset.

Note

The available sinks vary depending on target client, and within a target client, on other factors. For example, there is one set of sinks for “Points” node representation, and a different set of sinks for “Shapes” node representation. You can see the list of sinks by inspecting the defaultvisset.json file.

Note

The correctness of the functions provided in the func argument is not validated here. If the caller makes an error in the syntax of the function string, the errant string will still end up in the output visset.json. When the client loads the visset, the binding will throw an exception and the client will show an error message.

Returns:

True if the binding was set, or False if the sink could not be found.

Return type:

bool

Parameters:

• vis_section (str) – The visualization section of the visset.json file. For the Geospatial client, valid values for vis_section are “nodeVis” or “heatmapVis”. Other clients will have other visualization sections. See defaultvisset.json.

• sink_name (str) – The name of the sink to which to bind. E.g. “shapeExtrusion”.

• source_name (str) – The name of the data source to which to bind. E.g. “SpatialReport_Prevalence” or “BirthRate”. Can be None.

• func (str) –

  The function to use on the binding or None. The details of function syntax are documented elsewhere, but in general this may be “none()”, some built-in function e.g. “scale(3, 20)”, or a custom Javascript function body, e.g.:

  "{ return 1.0 - binding.value; }"
  

static opacity_one(weight, norm_weight)

add_weighted_network_layer(network, gradient_spec, layer_name='Network', layer_file_name='Network', opacity_func=None)

Adds a weighted network visualization layer to the output visset.

This method generates a CZML layer that provides a visual representation of a weighted network between nodes.

Note

Call set_products_directory and set_demographics before calling this method.

For:

Geospatial visualization.

Returns:

None. Creates a CZML file in the VisSet’s products directory.

Parameters:

• network (array) –

  array of objects:

  {
      from: <from-node-id>,
      to: <to-node-id>,
      weight: <float-weight>
  }
  

• gradient_spec (str) – gradient spec for a gradient with which to

• lines. (color the network) –

• layer_name (str) – Layer name (to be displayed in the client UI).l

• layer_file_name (str) – The layer base name for the output file. For

• example –

• "Network.czml". ("Network" results in an output file called) –

• opacity_func (function) – function(weight, norm_weight) that returns

• [0 (the desired opacity in range) –

• 1]. –

show_layer(vis_section)

Sets a layer to be shown by default in the output visset.json.

This method lets the caller set that the particular visualization section is shown by default when the visset is loaded by the client.

Returns:

None.

Parameters:

• vis_section (valid values for) – The visualization section of the visset.json

• client (file. For the Geospatial) –

• vis_section –

• other (are "nodeVis" or "heatmapVis". Other clients will have) –

• defaultvisset.json. (visualization sections. See) –

hide_layer(vis_section)

Sets a layer to be hidden by default in the output visset.json.

This method lets the caller set that the particular visualization section is hidden by default when the visset is loaded by the client. Note that all the data and bindings for that visualization section are still created and emitted to the visset…only their default visibility is affected.

Returns:

None.

Parameters:

• vis_section (valid values for) – The visualization section of the visset.json

• client (file. For the Geospatial) –

• vis_section –

• other (are "nodeVis" or "heatmapVis". Other clients will have) –

• defaultvisset.json. (visualization sections. See) –

exclude_spatial_channels(spatial_channel_names)

Exclude named channels from the list of binding sources.

Allows caller to exclude a set of spatial channels from showing up as sources for visualization bindings. This would primarily be done to reduce the memory pressure (or load time) of the client page, since all the source spatial binaries have to be loaded into memory on the client side. All this really does is set show=False in the link, which excludes the source on the client side.

Note

Call add_spatial_reports() before calling this method.

Returns:

None.

Parameters:

spatial_channel_names (list) –

a list or array of spatial channel names. If only one channel is to be excluded, enclose it in parens or [] so that it is iterable. E.g.:

exclude_spatial_channels(["SpatialReport_Adult_Vectors"])

include_spatial_channels(spatial_channel_names)

Include (only) named channels in the list of binding sources.

Allows caller to include a subset of the available spatial channels as the list of possible sources for visualization bindings. This would primarily be done to reduce the memory pressure (or load time) of the client page, since all the source spatial binaries have to be loaded into memory on the client side. All this really does is set the specified channels to show=True and all the rest to show=False.

Note

Call add_spatial_reports() before calling this method.

Returns:

None.

Parameters:

spatial_channel_names (list) –

a list or array of spatial channel names. If only one channel is to be excluded, enclose it in parens or [] so that it is iterable e. E.g.:

include_spatial_channels(["SpatialReport_Adult_Vectors"])

exclude_nodes_from_stats(node_id_array)

Excludes a set of nodes from node min/max statistics.

This method lets the caller specify a list of node ids that should be excluded from the min/max calculations data channels. This includes both static sources (e.g. NodeAttributes.InitialPopulation) and spatial reports (e.g. SpatialReport_Population). This is typically used to drop an outlier from the data set to avoid visually skewing the output visualization. For example, one might exclude the “elsewhere” node, which typically has extraordinary population, prevalence, etc. in a simulation that incorporates external migration.

Note

The node itself is not eliminated - it will still be visually represented, but it will have a value that may lie outside the min/ max range calculated for a given source. The client will clamp the value at visualization time.

Note

This method calls update_nodes…the caller need not do that.

Returns:

None.

Parameters:

node_id_array (list) –

a list or array of node id’s to be excluded. If only a single node is to be excluded, enclose it in parens or [] so that it is iterable, e.g.:

exclude_nodes_from_stats([10001])

or just call exclude_node_from_stats()

exclude_node_from_stats(node_id)

Exclude a node from node min/max statistics.

This method is exactly exclude_nodes_from_stats but for a single node id. See that method for details.

Returns:

None.

Parameters:

node_id (int) – Node id to exclude from stats.

update_nodes()

Updates the local summarized nodes from the Demographics source.

This method forces the VisSet object to refresh its summarized copy of the node data that it will later include in the written visset.json. Use this method if you change the data in the Demographics object and want those changes reflected in the output visset.json.

Returns:

None.

Parameters:

None. –

update_ranges()

Updates the local min/max values for extra node fields.

This method updates the local min/max values that are calculated and later emitted into the visset.json file. Use this method if you change the values in the Demographics object.

Returns:

None.

Parameters:

None. –

validate()

Validate the settings in the visset to ensure their validity.

This function validates the visset to increase the chances of it working correctly when brought into the client.

Returns:

None.

Parameters:

None. –

To do:

• Validate targetClient

• Validate nodeVis.visType

• Validate that self.nodes is populated

• Validate that self.timestepCount and startDate are populated

• Try to guess whether the list of spatial binaries is too big to be loaded on the client side

• Validate binding sources have show=true

• Validate binding functions to the extent possible

• Validate that the inset chart file is a JSON and contains the expected keys

write(sim_url_root=None)

Write the customized VisSet object to a visset.json file in the products directory.

This method is called after the caller has made all their additions and modifications to the default visset to write a visset.json file to the products directory.

Returns:

None. Writes a “visset.json” file to the VisSet’s products directory.

Parameters:

• sim_url_root (str) – Optional. Url root substitution if desired. If

• provided (not) –

• "./". (URLs in the output visset will begin with) –

Raises:

I/O and JSON exceptions. –

write_as(file_name, sim_url_root=None)

Write the customized VisSet object to a specified JSON file.

This method is called after the caller has made all their additions and modifications to the default visset to write a JSON file with the given name to the products directory.

Returns:

None. Writes a JSON file to the VisSet’s products directory.

Parameters:

• file_name (str) – The file name to use for the output visset JSON

• file. –

• sim_url_root (str) – Optional. Url root substitution if desired. If

• provided (not) –

• "./". (URLs in the output visset will begin with) –

Raises:

I/O and JSON exceptions. –

static update_urls(visset, asset_map_or_url_root)

Integrate COMPS Asset Manager URLs or update url roots.

This function does two different things, but they are encapsulated into this one function because this function is the only spot in VisSet that knows where all the URLs are in visset[links].

If passed an asset map, it adds “url_asset” entries next to “url” entries to allow the visset to be used in a hosted environment.

If passed a string, the “url” entries are updated to replace “./” with the given root. This makes the local paths suitable for use with a locally-hosted Vis-Tools.

Returns:

None.

Parameters:

• visset (obj) – A visset read with json.load or a VisSet __dict__.

• asset_map (obj|str) – Either an asset map read with json.load, or

• urls. (a string to be the new root for) –