idmtools_platform_comps.utils.ssmt_utils.file_filter module

idmtools ssmt file filter tools.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

idmtools_platform_comps.utils.ssmt_utils.file_filter.get_common_parser(app_description)

Creates a get common argument parser used with file filter function.

idmtools_platform_comps.utils.ssmt_utils.file_filter.gather_files(directory: str, file_patterns: List[str], exclude_patterns: Optional[List[str]] = None, assets: bool = False, prefix: Optional[str] = None, filename_format_func: Optional[Callable[[str], str]] = None) → Set[Tuple[str, str, UUID, int]]

Gather file_list.

Parameters:

• directory – Directory to gather from

• file_patterns – List of file patterns

• exclude_patterns – List of patterns to exclude

• assets – Should assets be included

• prefix – Prefix for file_list

• filename_format_func – Function that can format the filename

Returns:

Return files that match patterns.

idmtools_platform_comps.utils.ssmt_utils.file_filter.is_file_excluded(filename: str, exclude_patterns: List[str]) → bool

Is file excluded by excluded patterns.

Parameters:

• filename – File to filter

• exclude_patterns – List of file patterns to exclude

Returns:

True is file is excluded

idmtools_platform_comps.utils.ssmt_utils.file_filter.gather_files_from_related(work_item: WorkItem, file_patterns: List[str], exclude_patterns: List[str], assets: bool, simulation_prefix_format_str: str, work_item_prefix_format_str: str, entity_filter_func: Callable[[CommissionableEntity], bool], filename_format_func: Callable[[str], str]) → Set[Tuple[str, str, UUID, int]]

Gather files from different related entities.

Parameters:

• work_item – Work item to gather from

• file_patterns – List of File Patterns

• exclude_patterns – List of Exclude patterns

• assets – Should items be gathered from Assets Directory

• simulation_prefix_format_str – Format string for prefix of Simulations

• work_item_prefix_format_str – Format string for prefix of WorkItem

• entity_filter_func – Function to filter entities

• filename_format_func – Filename filter function

Returns:

Set of File Tuples in format Filename, Destination Name, and Checksum

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_experiments(assets: bool, entity_filter_func: Callable[[CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[Future], pool: ThreadPoolExecutor, simulation_prefix_format_str: str, work_item: WorkItem, filename_format_func: Callable[[str], str])

Filter Experiments outputs using our patterns.

Parameters:

• assets – Assets to filter

• entity_filter_func – Function to filter functions

• exclude_patterns_compiles – List of patterns to exclude

• file_patterns – File patterns to match

• futures – Future queue

• pool – Pool to execute jobs on

• simulation_prefix_format_str – Format string for prefix of Simulations

• work_item – Parent WorkItem

• filename_format_func – Function to filter filenames

Returns:

None

idmtools_platform_comps.utils.ssmt_utils.file_filter.get_simulation_prefix(parent_work_item: WorkItem, simulation: Simulation, simulation_prefix_format_str: str, experiment: Optional[Experiment] = None) → str

Get Simulation Prefix.

Parameters:

• parent_work_item – Parent workitem

• simulation – Simulation to form

• simulation_prefix_format_str – Prefix format string

• experiment – Optional experiment to be used with the

Returns:

Name of the simulation

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_experiment_assets(work_item: WorkItem, assets: bool, entity_filter_func: Callable[[CommissionableEntity], bool], exclude_patterns_compiles: List, experiment: Experiment, file_patterns: List[str], futures: List[Future], pool: ThreadPoolExecutor, simulation_prefix_format_str: str, simulations: List[Simulation], filename_format_func: Callable[[str], str])

Filter experiment assets. This method uses the first simulation to gather experiment assets.

Parameters:

• work_item – Parent Workitem

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• experiment – Experiment

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• simulation_prefix_format_str – Format string for simulation

• simulations – List of simulations

• filename_format_func – Name function for filename

Returns:

None

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_simulations_files(assets: bool, entity_filter_func: Callable[[CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[Future], pool: ThreadPoolExecutor, simulation_prefix_format_str: str, work_item: WorkItem, filename_format_func: Callable[[str], str])

Filter Simulations files.

Parameters:

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• simulation_prefix_format_str – Format string for simulation

• work_item –

• filename_format_func – Filename function

Returns:

None

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_simulation_list(assets: bool, entity_filter_func: Callable[[CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[Future], pool: ThreadPoolExecutor, simulation_prefix_format_str: str, simulations: List[Simulation], work_item: WorkItem, experiment: Optional[Experiment] = None, filename_format_func: Optional[Callable[[str], str]] = None)

Filter simulations list. This method is used for experiments and simulations.

Parameters:

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• simulation_prefix_format_str – Format string for simulation

• simulations – List of simulations

• work_item – Parent workitem

• experiment – Optional experiment.

• filename_format_func – Filename function

Returns:

None

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_work_items_files(assets: bool, entity_filter_func: Callable[[CommissionableEntity], bool], exclude_patterns_compiles: List, file_patterns: List[str], futures: List[Future], pool: ThreadPoolExecutor, work_item: WorkItem, work_item_prefix_format_str: str, filename_format_func: Callable[[str], str])

Filter work items files.

Parameters:

• assets – Whether assets should be matched

• entity_filter_func – Entity Filter Function

• exclude_patterns_compiles – List of files to exclude

• file_patterns – File patterns to filter

• futures – List of futures

• pool – Pool to submit search jobs to

• work_item – WorkItem

• work_item_prefix_format_str – WorkItemPrefix

• filename_format_func – Filename function

Returns:

None

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_ac_files(wi: WorkItem, patterns, exclude_patterns) → List[AssetCollectionFile]

Filter Asset Collection File.

Parameters:

• wi – WorkItem

• patterns – File patterns

• exclude_patterns – Exclude patterns

Returns:

List of filters asset collection files

idmtools_platform_comps.utils.ssmt_utils.file_filter.get_asset_file_path(file)

Get asset file path which combined the relative path and filename if relative path is set.

Otherwise we use just the filename.

Parameters:

file – Filename

Returns:

Filename

exception idmtools_platform_comps.utils.ssmt_utils.file_filter.DuplicateAsset

Bases: Exception

Error for when we encountered output paths that overlap.

doc_link: str = 'platforms/comps/assetize_output.html#errors'

idmtools_platform_comps.utils.ssmt_utils.file_filter.ensure_no_duplicates(ac_files, files)

Ensure no duplicates are in asset.

Parameters:

• ac_files – Ac files

• files – Simulation/Experiment/Workitem files

Returns:

None

Raises:

DuplicateAsset - if asset with same output path is found –

idmtools_platform_comps.utils.ssmt_utils.file_filter.print_results(ac_files, files)

Print Results.

Parameters:

• ac_files – Ac Files

• files – Files

Returns:

None

idmtools_platform_comps.utils.ssmt_utils.file_filter.apply_custom_filters(args: Namespace)

Apply user defined custom filter functions.

The function does the following workflow.

1. Check if there is a pre_run_func(s) defined. 1b) If there are pre-run funcs, run each of those 2) Is there an entity_filter_func. This function allows us to filter items(Experiment/Simulations/etc) directly. If not defined, we use a default function returns true. 3) If filename format function is defined, we set that, otherwise we use the default which just uses the original file name

Parameters:

args – argparse namespace.

Returns:

entity_filter_func and filename format func

idmtools_platform_comps.utils.ssmt_utils.file_filter.parse_filter_args_common(args: Namespace)

Parse filter arguments from an argparse namespace.

We need this because we use filtering across multiple scripts.

Parameters:

args – Argparse args

Returns:

entity_filter_func and filename formart func

idmtools_platform_comps.utils.ssmt_utils.file_filter.filter_files_and_assets(args: Namespace, entity_filter_func: Callable[[CommissionableEntity], bool], wi: WorkItem, filename_format_func: Callable[[str], str]) → Tuple[Set[Tuple[str, str, UUID, int]], List[AssetCollectionFile]]

Filter files and assets using provided parameters.

Parameters:

• args – Argparse details

• entity_filter_func – Optional filter function for entities. This function is ran on every item. If it returns true, we return the item

• wi – WorkItem we are running in

• filename_format_func – Filename format function allows use to customize how we filter filenames for output.

Returns:

Files that matches the filter and the assets that matches the filter as well.