paidiverpy.utils.parallellisation

Module for parallelisation utilities.

Functions

get_n_jobs(→ int)	Determine the number of jobs based on n_jobs parameter.
update_dask_config(→ None)	Update the Dask configuration.
parse_parallellisation_params(...)	Parse the client configuration.

Module Contents

paidiverpy.utils.parallellisation.get_n_jobs(n_jobs: int) → int

Determine the number of jobs based on n_jobs parameter.

Uses SLURM_CPUS_ON_NODE when inside a Slurm allocation so that only the CPUs actually allocated to the job are used, not all CPUs visible on the node.

Parameters:

n_jobs (int) – The number of n_jobs.

Returns:

The number of jobs to use.

Return type:

int

paidiverpy.utils.parallellisation.update_dask_config(dask_config_kwargs: dict) → None

Update the Dask configuration.

Parameters:

dask_config_kwargs (dict) – Dask configuration keyword arguments.

paidiverpy.utils.parallellisation.parse_parallellisation_params(config: paidiverpy.models.general_config.GeneralConfig | None) → dask.distributed.Client | None

Parse the client configuration.

Parameters:

config (GeneralConfig | None) – Client configuration.

Returns:

Dask client or None if no client is configured.

Return type:

dask.distributed.Client | None