deafrica_tools.dask

Functions for simplifying the creation of a local dask cluster.

Functions

create_dask_gateway_cluster(*args, **kwargs)

create_local_dask_cluster([spare_mem, …])

Using the datacube utils function ‘start_local_dask’, generate a local dask cluster.
deafrica_tools.dask.create_local_dask_cluster(spare_mem='3Gb', aws_unsigned=True, display_client=True, return_client=False, **kwargs)

Using the datacube utils function ‘start_local_dask’, generate a local dask cluster.

Example use :

from deafrica_tools.dask import create_local_dask_cluster

create_local_dask_cluster(spare_mem=’4Gb’)

Parameters

  • spare_mem (String, optional) – The amount of memory, in Gb, to leave for the notebook to run. This memory will not be used by the cluster. e.g ‘3Gb’

  • aws_unsigned (Bool, optional) – This parameter determines if credentials for S3 access are required and passes them on to processing threads, either local or on dask cluster. Set to True if working with publicly available datasets, and False if working with private data. i.e if loading Landsat C2 provisional data set this to aws_unsigned=False

  • display_client (Bool, optional) – An optional boolean indicating whether to display a summary of the dask client, including a link to monitor progress of the analysis. Set to False to hide this display.

  • return_client (Bool, optional) – An optional boolean indicating whether to return the dask client object.

  • **kwargs – Additional keyword arguments that will be passed to start_local_dask(). E.g. n_workers can be set to be greater than 1.