idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script module

exception idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.NoFileFound

Bases: Exception

doc_link: str = 'platforms/comps/assetize_output.html#errors'

exception idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.DuplicateAsset

Bases: Exception

doc_link: str = 'platforms/comps/assetize_output.html#errors'

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.gather_files(directory: str, file_patterns: List[str], exclude_patterns: List[str] = None, assets: bool = False, prefix: str = None) → Set[Tuple[str, str, uuid.UUID, int]]

Gather file_list

Parameters

• directory – Directory to gather from

• file_patterns – List of file patterns

• exclude_patterns – List of patterns to exclude

• assets – Should assets be included

• prefix – Prefix for file_list

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.is_file_excluded(filename: str, exclude_patterns: List[str]) → bool

Parameters

• filename – File to filter

• exclude_patterns – List of file patterns to exclude

Returns

True is file is excluded

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.gather_files_from_related(work_item: COMPS.Data.WorkItem.WorkItem, file_patterns: List[str], exclude_patterns: List[str], assets: bool, simulation_prefix_format_str: str, work_item_prefix_format_str: str, entity_filter_func: Callable[[COMPS.Data.CommissionableEntity.CommissionableEntity], bool]) → Set[Tuple[str, str, uuid.UUID, int]]

Gather files from different related entities

Parameters

• work_item – Work item to gather from

• file_patterns – List of File Patterns

• exclude_patterns – List of Exclude patterns

• assets – Should items be gathered from Assets Directory

• simulation_prefix_format_str – Format string for prefix of Simulations

• work_item_prefix_format_str – Format string for prefix of WorkItem

Returns

Set of File Tuples in format Filename, Destination Name, and Checksum

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.filter_experiments(assets: bool, entity_filter_func: Callable[[COMPS.Data.CommissionableEntity.CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[concurrent.futures._base.Future], pool: concurrent.futures.thread.ThreadPoolExecutor, simulation_prefix_format_str: str, work_item: COMPS.Data.WorkItem.WorkItem)

Filter Experiments outputs using our patterns

Parameters

• assets – Assets to filter

• entity_filter_func – Function to filter functions

• exclude_patterns_compiles – List of patterns to exclude

• file_patterns – File patterns to match

• futures – Future queue

• pool – Pool to execute jobs on

• simulation_prefix_format_str – Format string for prefix of Simulations

• work_item – Parent WorkItem

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.get_simulation_prefix(parent_workitem: COMPS.Data.WorkItem.WorkItem, simulation: COMPS.Data.Simulation.Simulation, simulation_prefix_format_str: str, experiment: COMPS.Data.Experiment.Experiment = None) → str

Get Simulation Prefix

Parameters

• parent_workitem – Parent workitem

• simulation – Simulation to form

• simulation_prefix_format_str – Prefix format string

• experiment – Optional experiment to be used with the

Returns

Name of the simulation

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.filter_experiment_assets(work_item: COMPS.Data.WorkItem.WorkItem, assets: bool, entity_filter_func: Callable[[COMPS.Data.CommissionableEntity.CommissionableEntity], bool], exclude_patterns_compiles: List, experiment: COMPS.Data.Experiment.Experiment, file_patterns: List[str], futures: List[concurrent.futures._base.Future], pool: concurrent.futures.thread.ThreadPoolExecutor, simulation_prefix_format_str: str, simulations: List[COMPS.Data.Simulation.Simulation])

Filter experiment assets. This method uses the first simulation to gather experiment assets

Parameters

• work_item – Parent Workitem

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• experiment – Experiment

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• simulation_prefix_format_str – Format string for simulation

• simulations – List of simulations

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.filter_simulations_files(assets: bool, entity_filter_func: Callable[[COMPS.Data.CommissionableEntity.CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[concurrent.futures._base.Future], pool: concurrent.futures.thread.ThreadPoolExecutor, simulation_prefix_format_str: str, work_item: COMPS.Data.WorkItem.WorkItem)

Filter Simulations files

Parameters

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• simulation_prefix_format_str – Format string for simulation

• work_item –

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.filter_simulation_list(assets: bool, entity_filter_func: Callable[[COMPS.Data.CommissionableEntity.CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[concurrent.futures._base.Future], pool: concurrent.futures.thread.ThreadPoolExecutor, simulation_prefix_format_str: str, simulations: List[COMPS.Data.Simulation.Simulation], work_item: COMPS.Data.WorkItem.WorkItem, experiment: COMPS.Data.Experiment.Experiment = None)

Filter simulations list. This method is used for experiments and simulations :param assets: Whether assets should be matched :param entity_filter_func: Entity Filter Function :param exclude_patterns_compiles: List of files to exclude :param file_patterns: File patterns to filter :param futures: List of futures :param pool: Pool to submit search jobs to :param simulation_prefix_format_str: Format string for simulation :param simulations: List of simulations :param work_item: Parent workitem :param experiment: Optional experiment.

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.filter_work_items_files(assets: bool, entity_filter_func: Callable[[COMPS.Data.CommissionableEntity.CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[concurrent.futures._base.Future], pool: concurrent.futures.thread.ThreadPoolExecutor, work_item: COMPS.Data.WorkItem.WorkItem, work_item_prefix_format_str: str)

Filter work items files

Parameters

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• work_item – WorkItem

• work_item_prefix_format_str – WorkItemPrefix

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.create_asset_collection(file_list: Set[Tuple[str, str, uuid.UUID, int]], ac_files: List[COMPS.Data.AssetCollectionFile.AssetCollectionFile], asset_tags: Dict[str, str])

Parameters

• file_list –

• asset_tags – Dict of tags to add to final asset

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.ensure_items_are_ready(work_item: COMPS.Data.WorkItem.WorkItem)

Ensure items are done. This leverages COMPS sleeping to pause creation of Asset Collections until item is in a done state :param work_item:

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.get_argument_parser()

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.assetize_error_handler(exctype, value: Exception, tb)

Global exception handler. This will write our errors in a nice format

Parameters

• exctype – Type of exception

• value – Value of the exception

• traceback – Traceback

Returns

None

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.filter_ac_files(patterns, exclude_patterns) → List[COMPS.Data.AssetCollectionFile.AssetCollectionFile]

Filter Asset Collection File

Parameters

• patterns – File patterns

• exclude_patterns – Exclude patterns

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.get_asset_file_path(file)

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.print_results(ac_files, files)

Print Results

Parameters

• ac_files – Ac Files

• files – Files

Returns:

idmtools_platform_comps.utils.assetize_output.assetize_ssmt_script.ensure_no_duplicates(ac_files, files)

Ensure no duplicates are in asset :param ac_files: Ac files :param files: Simulation/Experiment/Workitem files

Returns: