Container Platform#

Prerequisites#

  • Docker installed

  • Linux or Windows with WSL2

  • Python 3.8 or 3.8+ (https://www.python.org/downloads/release)

  • Create a Virtual Environment

    There are multiple ways to create a virtual environment. Below is an example using venv:

    python -m venv container_env

  • Activate Virtual Environment

    • On Windows:

    container_env\Scripts\activate

    • On Linux:

    source container_env/bin/activate

  • Install Container Platform

    pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple

ContainerPlatform#

The ContainerPlatform class is a part of the idmtools platform. This platform leverages Docker’s containerization capabilities to provide a consistent and isolated environment for running computational tasks. The ContainerPlatform is responsible for managing the creation, execution, and cleanup of Docker containers used to run simulations. It offers a high-level interface for interacting with Docker containers, allowing users to submit jobs, monitor their progress, and retrieve results. For more details on the architecture and the packages included in idmtools and ContainerPlatform, please refer to the documentation (Architecture and packages reference).

Key Features#

  • Docker Integration: Ensures that Docker is installed and the Docker daemon is running before executing any tasks.

  • Experiment and Simulation Management: Provides methods to run and manage experiments and simulations within Docker containers.

  • Volume Binding: Supports binding host directories to container directories, allowing for data sharing between the host and the container.

  • Container Validation: Validates the status and configuration of Docker containers to ensure they meet the platform’s requirements.

  • Script Conversion: Converts scripts to Linux format if the host platform is Windows, ensuring compatibility within the container environment.

  • Job History Management: Keeps track of job submissions and their corresponding container IDs for easy reference and management.

  • Minimal libraries and packages in Docker image: Requires only Linux os, python3, mpich installed in docker image. ContainerPlatform will bind the host directory to the container directory for running the simulation.

ContainerPlatform Attributes#

  • job_directory: The directory where job data is stored.

  • docker_image: The Docker image to run the container.

  • extra_packages: Additional packages to install in the container.

  • data_mount: The data mount point in the container.

  • user_mounts: User-defined mounts for additional volume bindings.

  • container_prefix: Prefix for container names.

  • force_start: Flag to force start a new container.

  • new_container: Flag to start a new container.

  • include_stopped: Flag to include stopped containers in operations.

  • debug: Flag to enable debug mode.

  • container_id: The ID of the container being used.

  • max_job: The maximum number of jobs to run in parallel.

  • retries: The number of retries to attempt for a job.

Usage#

The ContainerPlatform class is typically used to run computational experiments and simulations within Docker containers, ensuring a consistent and isolated environment. It provides various methods to manage and validate containers, submit jobs, and handle data volumes.

Example#

This example demonstrates how to use the ContainerPlatform class to run a simple command task within a Docker container.

Create a Python file named example.py on your host machine and add the following code:

from idmtools.entities.command_task import CommandTask
from idmtools.entities.experiment import Experiment
from idmtools_platform_container.container_platform import ContainerPlatform

# Initialize the platform
platform = ContainerPlatform(job_directory="destination_directory")
# OR
# from idmtools.core.platform_factory import Platform
# platform = Platform('Container', job_directory="destination_directory")

# Define task
command = "echo 'Hello, World!'"
task = CommandTask(command=command)
# Run an experiment
experiment = Experiment.from_task(task, name="example")
experiment.run(platform=platform)

To run example.py in the virtual environment on the host machine:

python example.py

You can find the simulation results in the job_directory/<suite_path>/<experiment_path>/<simulation_path> directory on the host machine.

Additionally, You can view the same results inside the Docker container at /home/container-data/<suite_path>/<experiment_path>/<simulation_path>.

More Examples#

Run the following included Python example to submit and run a job on your Container platform:

# This example demonstrates how to run a simulation using a container platform alias 'CONTAINER'.
import os
import sys
from functools import partial
from typing import Any, Dict
from idmtools.builders import SimulationBuilder
from idmtools.core.platform_factory import Platform
from idmtools.entities.experiment import Experiment
from idmtools.entities.simulation import Simulation
from idmtools.entities.templated_simulation import TemplatedSimulations
from idmtools_models.python.json_python_task import JSONConfiguredPythonTask

# job dir is where the experiment will be run.
# Define Container Platform. For full list of parameters see container_platform.py in idmtools_platform_container
platform = Platform("Container", job_directory="DEST")
# Define path to assets directory
assets_directory = os.path.join("..", "python_model", "inputs", "python", "Assets")
# Define task
task = JSONConfiguredPythonTask(script_path=os.path.join(assets_directory, "model.py"), parameters=(dict(c=0)))
task.python_path = "python3"
# Define templated simulation
ts = TemplatedSimulations(base_task=task)
ts.base_simulation.tags['tag1'] = 1
# Define builder
builder = SimulationBuilder()

# Define partial function to update parameter
def param_update(simulation: Simulation, param: str, value: Any) -> Dict[str, Any]:
    return simulation.task.set_parameter(param, value)

# Let's sweep the parameter 'a' for the values 0-2
builder.add_sweep_definition(partial(param_update, param="a"), range(3))

# Let's sweep the parameter 'b' for the values 0-4
builder.add_sweep_definition(partial(param_update, param="b"), range(5))
ts.add_builder(builder)

# Create Experiment using template builder
experiment = Experiment.from_template(ts, name="python example")
# Add our own custom tag to experiment
experiment.tags["tag1"] = 1
# And all files from assets_directory to experiment folder
experiment.assets.add_directory(assets_directory=assets_directory)

experiment.run(platform=platform, wait_until_done=True)
# run following command to check status
print("idmtools file DEST status --exp-id " + experiment.id)
sys.exit(0 if experiment.succeeded else -1)

Note

workitems and AssetCollection are not supported on the Container platform with idmtools. If you’ve used the COMPS platform with idmtools you may have scripts using these objects. You would need to update these scripts without using these objects in order to run them on the Container platform.