idmtools.utils.hashing module

Fast hash of Python objects.

Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.

class idmtools.utils.hashing.Hasher(hash_name='md5')

Bases: _Pickler

A subclass of pickler to do hashing, rather than pickling.

__init__(hash_name='md5')

Initialize our hasher.

Parameters:

hash_name – Hash type to use. Defaults to md5

hash(obj, return_digest=True)

Hash an object.

Parameters:

  • obj – Object to hash

  • return_digest – Should the digest be returned?

Returns:

None if return_digest is False, otherwise the hash digest is returned

save(obj)

Save an object to hash.

Parameters:

obj – Obj to save.

Returns:

None

memoize(obj)

Disable memoization for strings so hashing happens on value and not reference.

save_set(set_items)

Save set hashing.

Parameters:

set_items – Set items

Returns:

None

idmtools.utils.hashing.hash_obj(obj, hash_name='md5')

Quick calculation of a hash to identify uniquely Python objects.

Parameters:

  • obj – Object to hash

  • hash_name – The hashing algorithm to use. ‘md5’ is faster; ‘sha1’ is considered safer.

idmtools.utils.hashing.ignore_fields_in_dataclass_on_pickle(item)

Ignore certain fields for pickling on dataclasses.

Parameters:

item – Item to pickle

Returns:

State of item to pickle

idmtools.utils.hashing.calculate_md5(filename: str, chunk_size: int = 8192) → str

Calculate MD5.

Parameters:

  • filename – Filename to caclulate md5 for

  • chunk_size – Chunk size

Returns:

md5 as string

idmtools.utils.hashing.calculate_md5_stream(stream: Union[BytesIO, BinaryIO], chunk_size: int = 8192, hash_type: str = 'md5', file_hash=None)

Calculate md5 on stream.

Parameters:

  • chunk_size –

  • stream –

  • hash_type – Hash function

  • file_hash – File hash

Returns:

md5 of stream