idmtools_platform_comps.utils.package_version module#

idmtools Tools to filter versions of packages for requriements for asset collections.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

class idmtools_platform_comps.utils.package_version.PackageHTMLParser[source]#

Bases: HTMLParser, ABC

Base Parser for our other parsers.

previous_tag = None#
__init__()[source]#

Constructor.

pkg_version = None#
class idmtools_platform_comps.utils.package_version.LinkHTMLParser[source]#

Bases: PackageHTMLParser

Parse hrefs from links.

handle_starttag(tag, attrs)[source]#

Parse links and extra hrefs.

class idmtools_platform_comps.utils.package_version.LinkNameParser[source]#

Bases: PackageHTMLParser

Provides parsing of packages from pypi/arfifactory.

We parse links that match versions patterns

ver_pattern = re.compile('^[\\d\\.brcdev\\+nightly]+$')#
handle_starttag(tag, attrs)[source]#

Handle begin of links.

handle_endtag(tag)[source]#

End link tags.

handle_data(data)[source]#

Process links.

idmtools_platform_comps.utils.package_version.get_latest_package_version_from_pypi(pkg_name, display_all=False)[source]#

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • display_all – determine if output all package releases

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_latest_pypi_package_version_from_artifactory(pkg_name, display_all=False, base_version: str | None = None)[source]#

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • display_all – determine if output all package releases

  • base_version – Base version

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_pypi_package_versions_from_artifactory(pkg_name, display_all=False, base_version: str | None = None, exclude_pre_release: bool = True)[source]#

Utility to get versions of a package in artifactory.

Parameters:

  • pkg_name – package name given

  • display_all – determine if output all package releases

  • base_version – Base version

  • exclude_pre_release – Exclude any prerelease versions

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_latest_ssmt_image_version_from_artifactory(pkg_name='comps_ssmt_worker', base_version: str | None = None, display_all=False)[source]#

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • base_version – Optional base version. Versions above this will not be added.

  • display_all – determine if output all package releases

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_docker_manifest(image_path='idmtools/comps_ssmt_worker', repo_base='https://packages.idmod.org/artifactory/list/docker-production')[source]#

Get docker manifest from IDM Artifactory. It mimics latest even when user has no latest tag defined.

Parameters:

  • image_path – Path of docker image we want

  • repo_base – Base of the repo

Returns:

None

Raises:

ValueError - When the manifest cannot be found –

idmtools_platform_comps.utils.package_version.get_digest_from_docker_hub(repo, tag='latest')[source]#

Get the digest for image from docker.

Parameters:

  • repo – string, repository (e.g. ‘library/fedora’)

  • tag – string, tag of the repository (e.g. ‘latest’)

idmtools_platform_comps.utils.package_version.fetch_versions_from_server(pkg_url: str, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkHTMLParser'>) → List[str][source]#

Fetch all versions from server.

Parameters:

  • pkg_url – Url to fetch

  • parser – Parser tp use

Returns:

All the releases for a package

idmtools_platform_comps.utils.package_version.fetch_versions_from_artifactory(pkg_name: str, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkHTMLParser'>) → List[str][source]#

Fetch all versions from server.

Parameters:

  • pkg_name – Url to fetch

  • parser – Parser tp use

Returns:

Available releases

idmtools_platform_comps.utils.package_version.get_versions_from_site(pkg_url, base_version: str | None = None, display_all=False, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkNameParser'>, exclude_pre_release: bool = True)[source]#

Utility to get the the available versions for a package.

The default properties filter out pre releases. You can also include a base version to only list items starting with a particular version

Parameters:

  • pkg_url – package name given

  • base_version – Optional base version. Versions above this will not be added. For example, to get versions 1.18.5, 1.18.4, 1.18.3, 1.18.2 pass 1.18

  • display_all – determine if output all package releases

  • parser – Parser needs to be a HTMLParser that returns a pkg_versions

  • exclude_pre_release – Exclude prerelease versions

Returns: the latest version of ven package

Raises:

ValueError - If a latest versions cannot be determined –

idmtools_platform_comps.utils.package_version.get_latest_version_from_site(pkg_url, base_version: str | None = None, display_all=False, parser: ~typing.Type[~idmtools_platform_comps.utils.package_version.PackageHTMLParser] = <class 'idmtools_platform_comps.utils.package_version.LinkNameParser'>, exclude_pre_release: bool = True)[source]#

Utility to get the latest version for a given package name.

Parameters:

  • pkg_url – package name given

  • base_version – Optional base version. Versions above this will not be added.

  • display_all – determine if output all package releases

  • parser – Parser needs to be a HTMLParser that returns a pkg_versions

  • exclude_pre_release – Exclude pre-release versions

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.fetch_package_versions_from_pypi(pkg_name)[source]#

Utility to get the latest version for a given package name.

Parameters:

pkg_name – package name given

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.fetch_package_versions(pkg_name, is_released=True, sort=True, display_all=False)[source]#

Utility to get the latest version for a given package name.

Parameters:

  • pkg_name – package name given

  • is_released – get released version only

  • sort – make version sorted or not

  • display_all – determine if output all package releases

Returns: the latest version of ven package

idmtools_platform_comps.utils.package_version.get_highest_version(pkg_requirement: str)[source]#

Utility to get the latest version for a given package name.

Parameters:

pkg_requirement – package requirement given

Returns: the highest valid version of the package

idmtools_platform_comps.utils.package_version.get_latest_version(pkg_name)[source]#

Utility to get the latest version for a given package name.

Parameters:

pkg_name – package name given

Returns: the latest version of package

idmtools_platform_comps.utils.package_version.get_latest_compatible_version(pkg_name, base_version=None, versions=None, validate=True)[source]#

Utility to get the latest compatible version from a given version list.

Parameters:

  • base_version – Optional base version. Versions above this will not be added.

  • pkg_name – package name given

  • versions – user input of version list

  • validate – bool, if True, will validate base_version

Returns: the latest compatible version from versions

Raises:

Exception - If we cannot find version –

Notes

  • TODO - Make custom exception or use ValueError