idmtools_platform_local.infrastructure.service_manager module

idmtools service manager. Manages all our local platform services.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

class idmtools_platform_local.infrastructure.service_manager.DockerServiceManager(client: DockerClient, host_data_directory: str = '/home/docs/.local_data', network: str = 'idmtools', redis_image: str = 'redis:5.0.4-alpine', heartbeat_timeout: int = 15, redis_port: int = 6379, runtime: Optional[str] = 'runc', redis_mem_limit: str = '256m', redis_mem_reservation: str = '32m', postgres_image: str = 'postgres:11.4', postgres_mem_limit: str = '128m', postgres_mem_reservation: str = '32m', postgres_port: Optional[str] = 5432, workers_image: Optional[str] = None, workers_ui_port: int = 5000, workers_mem_limit: Optional[str] = None, workers_mem_reservation: str = '64m', run_as: Optional[str] = None, enable_singularity_support: bool = False, _services: Optional[Dict[str, BaseServiceContainer]] = None)

Bases: object

Provides single interface to manage all the local platform services.

client: DockerClient
host_data_directory: str = '/home/docs/.local_data'
network: str = 'idmtools'
redis_image: str = 'redis:5.0.4-alpine'
heartbeat_timeout: int = 15
redis_port: int = 6379
runtime: Optional[str] = 'runc'
redis_mem_limit: str = '256m'
redis_mem_reservation: str = '32m'
postgres_image: str = 'postgres:11.4'
postgres_mem_limit: str = '128m'
postgres_mem_reservation: str = '32m'
postgres_port: Optional[str] = 5432
workers_image: str = None
workers_ui_port: int = 5000
workers_mem_limit: str = None
workers_mem_reservation: str = '64m'
run_as: Optional[str] = None
enable_singularity_support: bool = False
init_services()

Start all the containers we should have running.

cleanup(delete_data: bool = False, tear_down_brokers: bool = False) → NoReturn

Stops the containers and removes the network.

Optionally the postgres data container can be deleted as well as closing any active Redis connections.

Parameters:

  • delete_data – Delete postgres data

  • tear_down_brokers – True to close redis brokers, false otherwise

Returns:

NoReturn

static setup_broker(heartbeat_timeout)

Start the broker to send data to workers.

static restart_brokers(heartbeat_timeout)

Restart brokers talking to workers.

create_services(spinner=None) → NoReturn

Create all the components of local platform.

Our architecture is as depicted in the UML diagram below

_images/58111b881264229fc93245b9a4037968cc5093cd27a1c506dc5351a77a0395ac.svg
Returns:

(NoReturn)

wait_on_ports_to_open(ports: List[str], wait_between_tries: Union[int, float] = 0.2, max_retries: int = 5, sleep_after: Union[int, float] = 0.5) → bool

Polls list of port attributes(eg postgres_port, redis_port and checks if they are currently open.

We use this to verify postgres/redis are ready for our workers

Parameters:

  • ports – List of port attributes

  • wait_between_tries – Time between port checks

  • max_retries – Max checks

  • sleep_after – Sleep after all our found open(Postgres starts accepting connections before actually ready)

Returns:

True if ports are ready

stop_services(spinner=None) → NoReturn

Stops all running IDM Tools services.

Returns:

(NoReturn)

get(container_name: str, create=True) → Container

Get the server with specified name.

Parameters:

  • container_name – Name of container

  • create – Create if it doesn’t exists

Returns:

Container

get_container_config(service: BaseServiceContainer, opts=None)

Get the container config for the service.

Parameters:

  • service – Service to get config for

  • opts – Opts to Extract. Should be a fields object

Returns:

Container config

restart_all(spinner=None) → NoReturn

Restart all the services IDM-Tools services.

Returns:

(NoReturn)

static is_port_open(host: str, port: int) → bool

Check if a port is open.

Parameters:

  • host – Host to check

  • port – Port to check

Returns:

True if port is open, False otherwise

static stop_service_and_wait(service) → bool

Stop server and wait.

Parameters:

service – Service to stop

Returns:

True

get_network() → Network

Fetches the IDM Tools network.

Returns:

(Network) Return Docker network object

__init__(client: DockerClient, host_data_directory: str = '/home/docs/.local_data', network: str = 'idmtools', redis_image: str = 'redis:5.0.4-alpine', heartbeat_timeout: int = 15, redis_port: int = 6379, runtime: Optional[str] = 'runc', redis_mem_limit: str = '256m', redis_mem_reservation: str = '32m', postgres_image: str = 'postgres:11.4', postgres_mem_limit: str = '128m', postgres_mem_reservation: str = '32m', postgres_port: Optional[str] = 5432, workers_image: Optional[str] = None, workers_ui_port: int = 5000, workers_mem_limit: Optional[str] = None, workers_mem_reservation: str = '64m', run_as: Optional[str] = None, enable_singularity_support: bool = False, _services: Optional[Dict[str, BaseServiceContainer]] = None) → None