idmtools_platform_local.infrastructure.docker_io module

idmtools docker input/output operations.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

class idmtools_platform_local.infrastructure.docker_io.DockerIO(host_data_directory: str = '/home/docs/.local_data')

Bases: object

Provides most of our file operations for our docker containers/local platform.

host_data_directory: str = '/home/docs/.local_data'
delete_files_below_level(directory, target_level=1, current_level=1)

Delete files below a certain depth in a target directory.

Parameters:

  • directory – Target directory

  • target_level – Depth to begin deleting. Default to 1

  • current_level – Current level. We call recursively so this should be 1 on initial call.

Returns:

None

Raises:

PermissionError - Raised if there are issues deleting a file. –

cleanup(delete_data: bool = True, shallow_delete: bool = False) → NoReturn

Stops the running services, removes local data, and removes network.

You can optionally disable the deleting of local data.

Parameters:

  • delete_data (bool) – When true, deletes local data

  • shallow_delete (bool) – Deletes the data but not the container folders(redis, workers). Preferred to preserve permissions and resolve docker issues

Returns:

(NoReturn)

copy_to_container(container: Container, destination_path: str, file: Optional[Union[str, bytes]] = None, content: Union[str, bytes] = None, dest_name: Optional[str] = None) → bool

Copies a physical file or content in memory to a container.

You can also choose a different name for the destination file by using the dest_name option.

Parameters:

  • container – Container to copy the file to

  • file – Path to the file to copy

  • content – Content to copy

  • destination_path – Path within the container to copy the file to(should be a directory)

  • dest_name – Optional parameter for destination filename. By default, the source filename is used

Returns:

(bool) True if the copy succeeds, False otherwise

sync_copy(futures)

Sync the copy operations queue in the io_queue.

This allows us to take advantage of multi-threaded copying while also making it convenient to have sync points, such as uploading the assets in parallel but pausing just before sync point.

Parameters:

futures – Futures to wait on.

Returns:

Final results of copy operations.

copy_multiple_to_container(container: Container, files: Dict[str, Dict[str, Any]], join_on_copy: bool = True)

Copy multiple items to a container.

Parameters:

  • container – Target container

  • files – Files to copy in form target directory -> filename -> data

  • join_on_copy – Should we join the threading on copy(treat as an atomic unit)

Returns:

True if copy succeeded, false otherwise

static create_archive_from_bytes(content: Union[bytes, BytesIO, BinaryIO], name: str) → BytesIO

Create a tar archive from bytes. Used to copy to docker.

Parameters:

  • content – Content to copy into tar

  • name – Name for file in archive

Returns:

(BytesIO) Return bytesIO object

create_directory(dir: str) → bool

Create a directory in a container.

Parameters:

dir – Path to directory to create

Returns:

(ExecResult) Result of the mkdir operation

__init__(host_data_directory: str = '/home/docs/.local_data') → None