deafrica_tools.dask#

Functions for simplifying the creation of a local dask cluster.

Functions

create_dask_gateway_cluster([profile, workers])

Create a cluster in our internal dask cluster.

create_local_dask_cluster([spare_mem, ...])

Using the datacube utils function start_local_dask, generate a local dask cluster.
deafrica_tools.dask.create_dask_gateway_cluster(profile='r5_L', workers=2)#

Create a cluster in our internal dask cluster.

Parameters:

  • profile (str) –

    Possible values are:

    • r5_L (2 cores, 15GB memory)

    • r5_XL (4 cores, 31GB memory)

    • r5_2XL (8 cores, 63GB memory)

    • r5_4XL (16 cores, 127GB memory)

  • workers (int) – Number of workers in the cluster.

deafrica_tools.dask.create_local_dask_cluster(spare_mem: str = '3Gb', display_client: bool = True, return_client: bool = False)#

Using the datacube utils function start_local_dask, generate a local dask cluster. Automatically detects if on AWS or NCI.

Parameters:

  • spare_mem (String, optional) – The amount of memory, in Gb, to leave for the notebook to run. This memory will not be used by the cluster. e.g ‘3Gb’

  • display_client (Bool, optional) – An optional boolean indicating whether to display a summary of the dask client, including a link to monitor progress of the analysis. Set to False to hide this display.

  • return_client (Bool, optional) – An optional boolean indicating whether to return the dask client object.