idmtools_platform_comps.utils.package_version module

idmtools Tools to filter versions of packages for requriements for asset collections.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

class idmtools_platform_comps.utils.package_version.PackageHTMLParser

Bases: HTMLParser, ABC

Base Parser for our other parsers.

previous_tag = None
__init__()

Constructor.

pkg_version = None
class idmtools_platform_comps.utils.package_version.LinkHTMLParser

Bases: PackageHTMLParser

Parse hrefs from links.

handle_starttag(tag, attrs)

Parse links and extra hrefs.

class idmtools_platform_comps.utils.package_version.LinkNameParser

Bases: PackageHTMLParser

Provides parsing of packages from pypi/arfifactory.

We parse links that match versions patterns

ver_pattern = re.compile('^[\\d\\.brcdev\\+nightly]+$')
handle_starttag(tag, attrs)

Handle begin of links.

handle_endtag(tag)

End link tags.

handle_data(data)

Process links.

idmtools_platform_comps.utils.package_version.get_latest_package_version_from_pypi(pkg_name, display_all=False)

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • display_all – determine if output all package releases

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_latest_pypi_package_version_from_artifactory(pkg_name, display_all=False, base_version: Optional[str] = None)

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • display_all – determine if output all package releases

  • base_version – Base version

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_pypi_package_versions_from_artifactory(pkg_name, display_all=False, base_version: Optional[str] = None, exclude_pre_release: bool = True)

Utility to get versions of a package in artifactory.

Parameters:

  • pkg_name – package name given

  • display_all – determine if output all package releases

  • base_version – Base version

  • exclude_pre_release – Exclude any prerelease versions

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_latest_ssmt_image_version_from_artifactory(pkg_name='comps_ssmt_worker', base_version: Optional[str] = None, display_all=False)

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • base_version – Optional base version. Versions above this will not be added.

  • display_all – determine if output all package releases

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_docker_manifest(image_path='idmtools/comps_ssmt_worker', repo_base='https://packages.idmod.org/artifactory/list/docker-production')

Get docker manifest from IDM Artifactory. It mimics latest even when user has no latest tag defined.

Parameters:

  • image_path – Path of docker image we want

  • repo_base – Base of the repo

Returns:

None

Raises:

ValueError - When the manifest cannot be found –

idmtools_platform_comps.utils.package_version.get_digest_from_docker_hub(repo, tag='latest')

Get the digest for image from docker.

Parameters:

  • repo – string, repository (e.g. ‘library/fedora’)

  • tag – string, tag of the repository (e.g. ‘latest’)

idmtools_platform_comps.utils.package_version.fetch_versions_from_server(pkg_url: str, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkHTMLParser'>) → List[str]

Fetch all versions from server.

Parameters:

  • pkg_url – Url to fetch

  • parser – Parser tp use

Returns:

All the releases for a package

idmtools_platform_comps.utils.package_version.fetch_versions_from_artifactory(pkg_name: str, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkHTMLParser'>) → List[str]

Fetch all versions from server.

Parameters:

  • pkg_name – Url to fetch

  • parser – Parser tp use

Returns:

Available releases

idmtools_platform_comps.utils.package_version.get_versions_from_site(pkg_url, base_version: ~typing.Optional[str] = None, display_all=False, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkNameParser'>, exclude_pre_release: bool = True)

Utility to get the the available versions for a package.

The default properties filter out pre releases. You can also include a base version to only list items starting with a particular version

Parameters:

  • pkg_url – package name given

  • base_version – Optional base version. Versions above this will not be added. For example, to get versions 1.18.5, 1.18.4, 1.18.3, 1.18.2 pass 1.18

  • display_all – determine if output all package releases

  • parser – Parser needs to be a HTMLParser that returns a pkg_versions

  • exclude_pre_release – Exclude prerelease versions

Returns: the latest version of ven package

Raises:

ValueError - If a latest versions cannot be determined –

idmtools_platform_comps.utils.package_version.get_latest_version_from_site(pkg_url, base_version: ~typing.Optional[str] = None, display_all=False, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkNameParser'>, exclude_pre_release: bool = True)

Utility to get the latest version for a given package name.

Parameters:

  • pkg_url – package name given

  • base_version – Optional base version. Versions above this will not be added.

  • display_all – determine if output all package releases

  • parser – Parser needs to be a HTMLParser that returns a pkg_versions

  • exclude_pre_release – Exclude pre-release versions

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.fetch_package_versions_from_pypi(pkg_name)

Utility to get the latest version for a given package name.

Parameters:

pkg_name – package name given

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.fetch_package_versions(pkg_name, is_released=True, sort=True, display_all=False)

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • is_released – get released version only

  • sort – make version sorted or not

  • display_all – determine if output all package releases

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_pkg_match_version(pkg_name, base_version=None, test='==', validate=True)

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • base_version – Optional base version. Versions above this will not be added.

  • test – default ==, a filter to find version

  • validate – bool, if True, will validate base_version

Returns: the latest version of ven package

Raises:

Exception - if we cannot find version –

Notes

  • TODO - Make custom exception or use ValueError

idmtools_platform_comps.utils.package_version.get_latest_version(pkg_name)

Utility to get the latest version for a given package name.

Parameters:

pkg_name – package name given

Returns: the latest version of package

Raises:

Exception if package could not be found. –

Notes

  • TODO - Make custom exception or use ValueError

idmtools_platform_comps.utils.package_version.get_latest_compatible_version(pkg_name, base_version=None, versions=None, validate=True)

Utility to get the latest compatible version from a given version list.

Parameters:

  • base_version – Optional base version. Versions above this will not be added.

  • pkg_name – package name given

  • versions – user input of version list

  • validate – bool, if True, will validate base_version

Returns: the latest compatible version from versions

Raises:

Exception - If we cannot find version –

Notes

  • TODO - Make custom exception or use ValueError