Enable parallel netCDF dataset reads#923
Enable parallel netCDF dataset reads#923davidhassell wants to merge 21 commits intoNCAS-CMS:mainfrom
Conversation
davidhassell
added
enhancement
sadielbartholomew
left a comment
sadielbartholomew
left a comment
There was a problem hiding this comment.
I'm using the appropriate cfdm branch (NCAS-CMS/cfdm#384) and have suitable dependencies etc. but there are immediate issues - should be easily solvable, though - see in-line comment.
sadielbartholomew
left a comment
sadielbartholomew
left a comment
There was a problem hiding this comment.
Tested as working and suitably tested downstream here in cf-python. A few minor comments in-line plus there is, ideally, the recipes linting/isort changes to revert (though I guess those only happened due to the historical / early 2025 initiation of this PR, since more recently I included the full recipes-docs/source/recipes in the linting isort skipping glob: https://github.com/NCAS-CMS/cf-python/blob/main/pyproject.toml#L25 so they won't auto-change any more).
Otherwise ready to merge, except for that we will update the class name for Netcdf_fileArray as discussed - I can do a final sanity check on both PRs once that change is made in both.
| * New optional backend for netCDF-3 in `cf.read` that allows parallel | ||
| reading: ``netcdf_file`` | ||
| (https://github.com/NCAS-CMS/cf-python/issues/912) | ||
| * Changed dependency: ``cfdm>=1.13.1.0, <1.13.2.0`` |
There was a problem hiding this comment.
This hasn't yet (here in this PR) been updated in the requirements.txt, but as long as it is done before release time that is fine.
| **2026-??-??** | ||
|
|
||
| * New default backend for netCDF-4 in `cf.read` that allows parallel | ||
| reading: (https://github.com/NCAS-CMS/cf-python/issues/912) |
There was a problem hiding this comment.
| reading: (https://github.com/NCAS-CMS/cf-python/issues/912) | |
| reading: ``h5netcdf-pyfive`` | |
| (https://github.com/NCAS-CMS/cf-python/issues/912) |
| from ...mixin_container import Container | ||
|
|
||
|
|
||
| class Netcdf_fileArray( |
There was a problem hiding this comment.
As discussed in the cfdm sibling PR, we shall rename this class and do so after my review so that there's no need for me to wait on the update here too. (A reminder.)
2 participants
Fixes #912
Requires NCAS-CMS/cfdm#384 to be resolved first.