-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance hit when using dask to compute missing_wmo
#1820
Comments
MWE and dask analysis: import xclim as xc
from xclim.testing import open_dataset
# Open a dataset of a single chunk
ds = open_dataset('sdba/CanESM2_1950-2100.nc', chunks={'time': -1, 'location': -1})
pr_valid = xc.core.missing.missing_wmo(ds.pr, freq="YS") The last line took me 115 s. But most importantly, counting the number of tasks with A probable solution would be to wrap as much as possible into single |
Adding a subissue: the Noted by @RondeauG . |
Setup Information
Description
I noticed a very significant performance hit when running
missing_wmo
with and without dask. The dask version seems to slow things down, and I had toload
the array to get results in a reasonable amount of time.Steps To Reproduce
No response
Additional context
No response
Contribution
Code of Conduct
The text was updated successfully, but these errors were encountered: