idmtools.analysis.csv_analyzer module

class idmtools.analysis.csv_analyzer.CSVAnalyzer(filenames, parse=True)

Bases: idmtools.entities.ianalyzer.IAnalyzer

Provides an analyzer for CSV output

Examples

Simple Example

This example covers the basic usage of the CSVAnalyzer

# Example CSVAnalyzer for any experiment
# In this example, we will demonstrate how to use a CSVAnalyzer to analyze csv files for experiments

# First, import some necessary system and idmtools packages.
from logging import getLogger

from idmtools.analysis.analyze_manager import AnalyzeManager
from idmtools.analysis.csv_analyzer import CSVAnalyzer
from idmtools.core import ItemType
from idmtools.core.platform_factory import Platform


if __name__ == '__main__':

    # Set the platform where you want to run your analysis
    # In this case we are running in COMPS since the Work Item we are analyzing was run on COMPS
    logger = getLogger()
    with Platform('COMPS2') as platform:

        # Arg option for analyzer init are uid, working_dir, data in the method map (aka select_simulation_data),
        # and filenames
        # In this case, we want to provide a filename to analyze
        filenames = ['output/c.csv']
        # Initialize the analyser class with the path of the output csv file
        analyzers = [CSVAnalyzer(filenames=filenames)]

        # Set the experiment id you want to analyze
        experiment_id = '9311af40-1337-ea11-a2be-f0921c167861'  # staging exp id simple sim and csv example

        # Specify the id Type, in this case an Experiment on COMPS
        manager = AnalyzeManager(configuration={}, partial_analyze_ok=True, ids=[(experiment_id, ItemType.EXPERIMENT)],
                                 analyzers=analyzers)
        manager.analyze()

Multiple CSVs

This example covers analyzing multiple CSVs

# Example CSVAnalyzer for any experiment with multiple csv outputs
# In this example, we will demonstrate how to use a CSVAnalyzer to analyze csv files for experiments

# First, import some necessary system and idmtools packages.
from idmtools.analysis.analyze_manager import AnalyzeManager
from idmtools.analysis.csv_analyzer import CSVAnalyzer
from idmtools.core import ItemType
from idmtools.core.platform_factory import Platform


if __name__ == '__main__':

    # Set the platform where you want to run your analysis
    # In this case we are running in COMPS since the Work Item we are analyzing was run on COMPS
    platform = Platform('COMPS2')

    # Arg option for analyzer init are uid, working_dir, data in the method map (aka select_simulation_data),
    # and filenames
    # In this case, we have multiple csv files to analyze
    filenames = ['output/a.csv', 'output/b.csv']
    # Initialize the analyser class with the path of the output csv file
    analyzers = [CSVAnalyzer(filenames=filenames)]

    # Set the experiment id you want to analyze
    experiment_id = '1bddce22-0c37-ea11-a2be-f0921c167861'  # staging exp id with multiple csv file outputs

    # Specify the id Type, in this case an Experiment on COMPS
    manager = AnalyzeManager(configuration={}, partial_analyze_ok=True, platform=platform,
                             ids=[(experiment_id, ItemType.EXPERIMENT)],
                             analyzers=analyzers)
    manager.analyze()

initialize()

Call once after the analyzer has been added to the AnalyzeManager.

Add everything depending on the working directory or unique ID here instead of in __init__.

map(data, simulation)

In parallel for each simulation, consume raw data from filenames and emit selected data.

Parameters

• data – A dictionary associating filename with content for simulation data.

• item – IItem object that the passed data is associated with.

Returns

Selected data for the given item.

reduce(all_data)

Combine the map() data for a set of items into an aggregate result.

Parameters

all_data – A dictionary with entries for the item ID and selected data.