idmtools.analysis.analyze_manager module

idmtools Analyzer manager.

AnalyzerManager is the “driver” of analysis. Analysis is mostly a map reduce operation.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

idmtools.analysis.analyze_manager.pool_worker_initializer(func, analyzers, platform: IPlatform) → NoReturn

Initialize the pool worker, which allows the process pool to associate the analyzers, cache, and path mapping to the function executed to retrieve data.

Using an initializer improves performance.

Parameters:

  • func – The function that the pool will call.

  • analyzers – The list of all analyzers to run.

  • platform – The platform to communicate with to retrieve files from.

Returns:

None

class idmtools.analysis.analyze_manager.AnalyzeManager(platform: IPlatform = None, configuration: dict = None, ids: List[Tuple[Union[str, uuid.UUID], idmtools.core.enums.ItemType]] = None, analyzers: List[idmtools.entities.ianalyzer.IAnalyzer] = None, working_dir: str = None, partial_analyze_ok: bool = False, max_items: Optional[int] = None, verbose: bool = True, force_manager_working_directory: bool = False, exclude_ids: List[Union[str, uuid.UUID]] = None, analyze_failed_items: bool = False, max_workers: Optional[int] = None, executor_type: str = 'process')

Bases: object

Analyzer Manager Class. This is the main driver of analysis.

ANALYZE_TIMEOUT = 28800
WAIT_TIME = 1.15
EXCEPTION_KEY = '__EXCEPTION__'
exception TimeOutException

Bases: Exception

TimeOutException is raised when the analysis times out.

exception ItemsNotReady

Bases: Exception

ItemsNotReady is raised when items to be analyzed are still running.

Notes

TODO - Add doc_link

__init__(platform: IPlatform = None, configuration: dict = None, ids: List[Tuple[Union[str, uuid.UUID], idmtools.core.enums.ItemType]] = None, analyzers: List[idmtools.entities.ianalyzer.IAnalyzer] = None, working_dir: str = None, partial_analyze_ok: bool = False, max_items: Optional[int] = None, verbose: bool = True, force_manager_working_directory: bool = False, exclude_ids: List[Union[str, uuid.UUID]] = None, analyze_failed_items: bool = False, max_workers: Optional[int] = None, executor_type: str = 'process')

Initialize the AnalyzeManager.

Parameters:

  • platform (IPlatform) – Platform

  • configuration (dict, optional) – Initial Configuration. Defaults to None.

  • ids (Tuple[UUID, ItemType], optional) – List of ids as pair of Tuple and ItemType. Defaults to None.

  • analyzers (List[IAnalyzer], optional) – List of Analyzers. Defaults to None.

  • working_dir (str, optional) – The working directory. Defaults to os.getcwd().

  • partial_analyze_ok (bool, optional) – Whether partial analysis is ok. When this is True, Experiments in progress or Failed can be analyzed. Defaults to False.

  • max_items (int, optional) – Max Items to analyze. Useful when developing and testing an Analyzer. Defaults to None.

  • verbose (bool, optional) – Print extra information about analysis. Defaults to True.

  • force_manager_working_directory (bool, optional) – [description]. Defaults to False.

  • exclude_ids (List[UUID], optional) – [description]. Defaults to None.

  • analyze_failed_items (bool, optional) – Allows analyzing of failed items. Useful when you are trying to aggregate items that have failed. Defaults to False.

  • max_workers (int, optional) – Set the max workers. If not provided, falls back to the configuration item max_threads. If max_workers is not set in configuration, defaults to CPU count

  • executor_type – (str): Whether to use process or thread pooling. Process pooling is more efficient but threading might be required in some environments

add_item(item: idmtools.core.interfaces.ientity.IEntity) → NoReturn

Add an additional item for analysis.

Parameters:

item – The new item to add for analysis.

Returns:

None

add_analyzer(analyzer: idmtools.entities.ianalyzer.IAnalyzer) → NoReturn

Add another analyzer to use on the items to be analyzed.

Parameters:

analyzer – An analyzer object (IAnalyzer).

Returns:

None

analyze() → bool

Process the provided items with the provided analyzers. This is the main driver method of AnalyzeManager.

Parameters:

kwargs – extra parameters

Returns:

True on success; False on failure/exception.