idmtools_platform_comps.utils.download.download module

idmtools download work item output.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

class idmtools_platform_comps.utils.download.download.CompressType(value)[source]

Bases: Enum

Defines the compression types we support.

lzma is the best balance between speed and compression ratio typically

lzma = 'lzma'
deflate = 'deflate'
bz = 'bz'
class idmtools_platform_comps.utils.download.download.DownloadWorkItem(_uid: str = None, _IItem__pre_creation_hooks: ~typing.List[~typing.Callable[[IItem, IPlatform], None]] = <factory>, _IItem__post_creation_hooks: ~typing.List[~typing.Callable[[IItem, IPlatform], None]] = <factory>, platform_id: str = None, _platform: IPlatform = None, parent_id: str = None, _parent: IEntity = None, status: ~idmtools.core.enums.EntityStatus = None, tags: ~typing.Dict[str, ~typing.Any] = <factory>, _platform_object: ~typing.Any = None, _IRunnableEntity__pre_run_hooks: ~typing.List[~typing.Callable[[IRunnableEntity, IPlatform], None]] = <factory>, _IRunnableEntity__post_run_hooks: ~typing.List[~typing.Callable[[IRunnableEntity, IPlatform], None]] = <factory>, name: str = 'idmtools workflow item', assets: ~idmtools.assets.asset_collection.AssetCollection = <factory>, item_name: dataclasses.InitVar[str] = None, asset_collection_id: dataclasses.InitVar[str] = None, transient_assets: ~idmtools.assets.asset_collection.AssetCollection = <factory>, asset_files: dataclasses.InitVar[FileList] = None, user_files: dataclasses.InitVar[FileList] = None, task: ~idmtools.entities.itask.ITask = None, related_experiments: list = <factory>, related_simulations: list = <factory>, related_suites: list = <factory>, related_work_items: list = <factory>, related_asset_collections: list = <factory>, work_item_type: str = None, work_order: dict = <factory>, plugin_key: str = '1.0.0.0_RELEASE', docker_image: str = None, command: dataclasses.InitVar[str] = None, file_patterns: ~typing.List[str] = <factory>, exclude_patterns: ~typing.List[str] = <factory>, include_assets: bool = False, simulation_prefix_format_str: str = '{simulation.id}', work_item_prefix_format_str: str = None, no_simulation_prefix: bool = False, verbose: bool = False, pre_run_functions: ~typing.List[~typing.Callable] = <factory>, entity_filter_function: ~typing.Callable[[~COMPS.Data.CommissionableEntity.CommissionableEntity], bool] = None, filename_format_function: ~typing.Callable[[str], str] = None, dry_run: bool = False, _ssmt_script: str = None, _ssmt_depends: ~typing.List[str] = <factory>, output_path: str = <factory>, extract_after_download: bool = True, delete_after_download: bool = True, zip_name: str = 'output.zip', compress_type: ~idmtools_platform_comps.utils.download.download.CompressType = None)[source]

Bases: FileFilterWorkItem

DownloadWorkItem provides a utility to download items through a workitem with compression.

The main advantage of this over Analyzers is the compression. This is most effective when the targets to download have many items that are similar to download. For example, an experiment with 1000 simulations with similar output can greatly benefit from downloading through this method.

Notes

  • TODO Link examples here.

output_path: str
extract_after_download: bool = True
delete_after_download: bool = True
zip_name: str = 'output.zip'
compress_type: CompressType = None
wait(wait_on_done_progress: bool = True, timeout: int | None = None, refresh_interval=None, platform: IPlatform | None = None) → None[source]

Waits on Download WorkItem to finish. This first waits on any dependent items to finish(Experiment/Simulation/WorkItems).

Parameters:

  • wait_on_done_progress – When set to true, a progress bar will be shown from the item

  • timeout – Timeout for waiting on item. If none, wait will be forever

  • refresh_interval – How often to refresh progress

  • platform – Platform

Returns:

AssetCollection created if item succeeds

__init__(_uid: str = None, _IItem__pre_creation_hooks: ~typing.List[~typing.Callable[[IItem, IPlatform], None]] = <factory>, _IItem__post_creation_hooks: ~typing.List[~typing.Callable[[IItem, IPlatform], None]] = <factory>, platform_id: str = None, _platform: IPlatform = None, parent_id: str = None, _parent: IEntity = None, status: ~idmtools.core.enums.EntityStatus = None, tags: ~typing.Dict[str, ~typing.Any] = <factory>, _platform_object: ~typing.Any = None, _IRunnableEntity__pre_run_hooks: ~typing.List[~typing.Callable[[IRunnableEntity, IPlatform], None]] = <factory>, _IRunnableEntity__post_run_hooks: ~typing.List[~typing.Callable[[IRunnableEntity, IPlatform], None]] = <factory>, name: str = 'idmtools workflow item', assets: ~idmtools.assets.asset_collection.AssetCollection = <factory>, item_name: dataclasses.InitVar[str] = None, asset_collection_id: dataclasses.InitVar[str] = None, transient_assets: ~idmtools.assets.asset_collection.AssetCollection = <factory>, asset_files: dataclasses.InitVar[FileList] = None, user_files: dataclasses.InitVar[FileList] = None, task: ~idmtools.entities.itask.ITask = None, related_experiments: list = <factory>, related_simulations: list = <factory>, related_suites: list = <factory>, related_work_items: list = <factory>, related_asset_collections: list = <factory>, work_item_type: str = None, work_order: dict = <factory>, plugin_key: str = '1.0.0.0_RELEASE', docker_image: str = None, command: dataclasses.InitVar[str] = None, file_patterns: ~typing.List[str] = <factory>, exclude_patterns: ~typing.List[str] = <factory>, include_assets: bool = False, simulation_prefix_format_str: str = '{simulation.id}', work_item_prefix_format_str: str = None, no_simulation_prefix: bool = False, verbose: bool = False, pre_run_functions: ~typing.List[~typing.Callable] = <factory>, entity_filter_function: ~typing.Callable[[~COMPS.Data.CommissionableEntity.CommissionableEntity], bool] = None, filename_format_function: ~typing.Callable[[str], str] = None, dry_run: bool = False, _ssmt_script: str = None, _ssmt_depends: ~typing.List[str] = <factory>, output_path: str = <factory>, extract_after_download: bool = True, delete_after_download: bool = True, zip_name: str = 'output.zip', compress_type: ~idmtools_platform_comps.utils.download.download.CompressType = None) → None