Fix CMIP7 variables written as double instead of float, causing fill value precision drift - #570
Merged
Merged
Conversation
…ion drift CMIP7 CMOR tables don't set a per-variable "type" (unlike CMIP6), so the dtype-casting fallback in the sea-ice/ocean/atmosphere CMORisers always hit its hardcoded "double" default, upcasting every CMIP7 variable to float64 regardless of the source data's actual precision. That upcast also happened after _FillValue/missing_value were computed against the pre-cast dtype, so the sentinel value silently drifted from the data's fill cells once promoted to float64 (float32(1e20) != 1e20 after promotion), breaking exact-equality masking downstream. Fall back to the source array's own dtype instead of hardcoding double, re-cast fill/missing value attrs to the final dtype whenever a cast does happen, and switch to tolerance-based fill-value masking in the iris conversion path as a safety net. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #570 +/- ##
=======================================
+ Coverage 74.9% 75.0% +0.1%
=======================================
Files 36 36
Lines 7503 7517 +14
Branches 1409 1415 +6
=======================================
+ Hits 5616 5634 +18
+ Misses 1589 1586 -3
+ Partials 298 297 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…butes() Adds regression tests for the CMIP7 dtype fallback fix: no per-variable type preserves the source dtype, an explicit double upcasts and re-casts _FillValue/missing_value to match, and valid_min/valid_max are cast to the same target dtype before range checking. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Member
Author
|
Fix #569 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
siconc, and likely most/all CMIP7 output variables, were being written asdoubleeven though the source model data isfloat32. Two problems:float32, then the data gets upcast todoubleafterward - but the fill value attribute doesn't get upcast with it.float32(1e20)promoted todoubleis1.0000000200408773e+20, not exactly1e20, so exact-equality masking against_FillValuesilently fails and masked cells show up as spikes instead of being masked out in plots.Root cause: the CMORiser picks output dtype from the CMOR table's
"type"field, defaulting to"double"when missing. CMIP7 tables never set"type"per-variable (CMIP6's do), so every CMIP7 variable fell through to that default regardless of the source dtype.Changes
sea_ice.py,ocean.py,atmosphere.py: default to the source array's own dtype instead of hardcodingdouble; re-cast_FillValue/missing_valueto the final dtype whenever a cast does happen.driver.py: switched fill-value masking in the iris conversion path (to_iris()) from exact equality to tolerance-based comparison, as a safety net.Test plan
pixi run -e test pytest tests/unit/- 1552 passed, 2 skippedsiconcoutput file to confirm it's nowfloatand masks correctly in plots