Repository navigation
Read channel names from the "channel" dataset in h5_convert.py - #183
Merged
azrael417 merged 1 commit intoSep 30, 2026
Merged
Conversation
Every makani writer, h5_convert.py included, stores channel names in a dataset called "channel", so reading "channel_names" raised a KeyError on every annotated file. Add a test that converts an annotated dataset.
azrael417
approved these changes
Sep 30, 2026
azrael417
left a comment
azrael417
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, thanks for fixing this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
data_process/h5_convert.pyreads the channel names from a dataset calledchannel_names, but every makani writer (annotate_dataset.py,concatenate_dataset.py, the converters, andh5_convert.pyitself when it writes its output) calls that datasetchannel. So the script stops withKeyError: "Unable to synchronously open object (object 'channel_names' doesn't exist)"on any annotated file and writes nothing. This changes the read tochannel.I also added
TestH5Converttotests/test_data_process.py. It annotates a small dataset with the existing test helpers, converts it with gzip and a batch size that leaves a partial last batch, and checks the data, the compression and the four dimension scales. It fails with the KeyError on main and passes with the fix.Testing:
tests/test_data_process.pypasses locally (11 passed, 8 skipped for mpi4py, the same skips as on main),pre-commit run --all-filesis clean, and thetestsandStyle checksworkflows passed on my fork (tests, style). I also ran the script from the command line on an annotated file before and after the change. I only tried small synthetic files, not real ERA5 data.Two other options of the script still fail for unrelated reasons (
--transposeand a chunk size given in MB). I left them alone to keep this small.