Add keyword to allow disabling config forwarding in SSHCluster #8994
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #8138
#8138 shows that in Windows there is a limit to environment variable size, and
SSHCluster
serialises your Dask config to forward to the remote processes. If your Dask config is really big then this will fail. There isn't a clear solution here, in #4465 we explored compressing config, but that isn't a perfect solution so hasn't been merged.A reasonable workaround is to be able to just disable this functionality if you don't need it. Users in #8138 (comment) have reported just modifying
distributed
to disable this. This PR adds a kwarg to officially support disabling config forwarding.pre-commit run --all-files