Assetize outputs

For an overview of assetizing outputs, see Assetize outputs workitem.

Excluding files from assetizing

Sometimes some of the files overlap with patterns you would like to include in a destination. In the following example, there are 100 files created with the .out extension. We would like all the files except “3.out” and “5.out”. We can just add these two files to the exclude patterns of an AssetizeOutput object.

from idmtools.core.platform_factory import Platform
from idmtools.entities.command_task import CommandTask
from idmtools.entities.experiment import Experiment
from idmtools_platform_comps.utils.assetize_output.assetize_output import AssetizeOutput

task = CommandTask(command="python Assets/model.py")
task.common_assets.add_asset("model.py")

platform = Platform("CALCULON")
experiment = Experiment.from_task(task)

# Since we have one simulation in our experiment, we can "flatten output" by using the format str
ao = AssetizeOutput(file_patterns=["*.out"], related_experiments=[experiment], no_simulation_prefix=True)
# Exclude some output files while preserving the default exclusions of stdout and stderr.txt
ao.exclude_patterns.append("3.out")
ao.exclude_patterns.append("5.out")
ao.run(wait_until_done=True)

if ao.succeeded:
    for asset in sorted(ao.asset_collection, key=lambda sa: sa.short_remote_path().rjust(6)):
        print(asset.short_remote_path().rjust(6))
else:
    print('Item failed. Check item output')

Using with experiments

The following demonstrates assetizing the output of experiments. An important part to remember with experiments is that they typically have multiple simulations. To avoid conflicts in the assetized output, the default behavior is to use the simulation.id as a folder name for each simulation output. We can include the original experiment asset collection in filtering as well by using the include_assets parameter.

from functools import partial
from idmtools.builders import SimulationBuilder
from idmtools.core.platform_factory import Platform
from idmtools.entities.command_task import CommandTask
from idmtools.entities.experiment import Experiment
from idmtools_platform_comps.utils.assetize_output.assetize_output import AssetizeOutput


############## Setup outputs to assetize in demo

base_task = CommandTask(command="python3 model.py")
base_task.common_assets.add_asset("model.py")
# Command task have no configs. Since it is a python object, we add our own item
base_task.config = dict(a=1,b=1)

# define a template for our commands
command_template = "python Assets/model.py --a {a} --b {b}"


# Define a function that renders our command line as we build simulations
def create_command_line_hook(simulation, platform):
    # we get our simulations object. Use the task and render our command line
    simulation.task.command = command_template.format(**simulation.task.config)


# Define sweeps
def set_parameter(simulation, parameter, value):
    simulation.task.config[parameter] = value


# add hook to our base task
base_task.add_pre_creation_hook(create_command_line_hook)
builder = SimulationBuilder()
builder.add_sweep_definition(partial(set_parameter, parameter="a"), range(3))
builder.add_sweep_definition(partial(set_parameter, parameter="b"), range(3))

platform = Platform("CALCULON")

experiment = Experiment.from_builder(builders=builder, base_task=base_task, name="Create example output")

############### Demo of assetize of experiment outputs
# Define what we want to assetize output
ao = AssetizeOutput(file_patterns=["output.json"], related_experiments=[experiment])
# run the Assetize job. It will ensure other items are ran if they are entities(Experiment, Simulation or Workitems)
ao.run(wait_until_done=True)
print(f"Asset Collection: {ao.id}")