Multiday download from ECMWF CDS API - #810
Draft
jlmaurer wants to merge 21 commits into
Draft
Conversation
8 tasks
Collaborator
Author
|
Replaces #802 |
jlmaurer
added a commit
to jlmaurer/RAiDER-1
that referenced
this pull request
Aug 11, 2026
…tests The single combined request for lnsp/z/t/q could not work. ERA-5 archives lnsp and z at model level 1 only, so a mixed-level request is not something CDS will return as one netCDF -- the multi-date path added in dbekaert#810 splits the request for exactly this reason. Where a single file is returned at all, the surface fields come back padded onto the full 137-level axis, so .squeeze() no longer collapses the level dimension and calcgeoh raises: ValueError: could not broadcast input array from shape (137,11,11) into (11,11) reproduced against test/weather_files/ERA-5_2020_01_30_T13_52_45.nc, which was downloaded by the pre-split single-request path and still carries that layout. Restore the two requests and keep the rest of the cleanup: the full-cube shutil.copy is dead (to_netcdf overwrites out_path) and is dropped. Nothing in CI performs a CDS retrieval, which is why _get_from_cds had no cover at all. test_ecmwf_fetch.py stubs cdsapi.Client and drives the real post-processing into the real reader, asserting the layout contract the two share: t/q/z as (time, level, lat, lon) cubes and lnsp recovered as lnsp[0, 0]. The stub refuses a mixed-level request, so re-merging the two retrievals fails the suite instead of failing at download time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ERA-5 model-level data was fetched in two separate CDS calls (lnsp+z, then t+q) and recombined on disk via a temp-file copy. Request all four params in one retrieval instead: one queue wait instead of two, one temp file instead of three, and no shutil.copy of the full cube.
jlmaurer
added a commit
to jlmaurer/RAiDER-1
that referenced
this pull request
Aug 11, 2026
…tests The single combined request for lnsp/z/t/q could not work. ERA-5 archives lnsp and z at model level 1 only, so a mixed-level request is not something CDS will return as one netCDF -- the multi-date path added in dbekaert#810 splits the request for exactly this reason. Where a single file is returned at all, the surface fields come back padded onto the full 137-level axis, so .squeeze() no longer collapses the level dimension and calcgeoh raises: ValueError: could not broadcast input array from shape (137,11,11) into (11,11) reproduced against test/weather_files/ERA-5_2020_01_30_T13_52_45.nc, which was downloaded by the pre-split single-request path and still carries that layout. Restore the two requests and keep the rest of the cleanup: the full-cube shutil.copy is dead (to_netcdf overwrites out_path) and is dropped. Nothing in CI performs a CDS retrieval, which is why _get_from_cds had no cover at all. test_ecmwf_fetch.py stubs cdsapi.Client and drives the real post-processing into the real reader, asserting the layout contract the two share: t/q/z as (time, level, lat, lon) cubes and lnsp recovered as lnsp[0, 0]. The stub refuses a mixed-level request, so re-merging the two retrievals fails the suite instead of failing at download time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tests The single combined request for lnsp/z/t/q could not work. ERA-5 archives lnsp and z at model level 1 only, so a mixed-level request is not something CDS will return as one netCDF -- the multi-date path added in dbekaert#810 splits the request for exactly this reason. Where a single file is returned at all, the surface fields come back padded onto the full 137-level axis, so .squeeze() no longer collapses the level dimension and calcgeoh raises: ValueError: could not broadcast input array from shape (137,11,11) into (11,11) reproduced against test/weather_files/ERA-5_2020_01_30_T13_52_45.nc, which was downloaded by the pre-split single-request path and still carries that layout. Restore the two requests and keep the rest of the cleanup: the full-cube shutil.copy is dead (to_netcdf overwrites out_path) and is dropped. Nothing in CI performs a CDS retrieval, which is why _get_from_cds had no cover at all. test_ecmwf_fetch.py stubs cdsapi.Client and drives the real post-processing into the real reader, asserting the layout contract the two share: t/q/z as (time, level, lat, lon) cubes and lnsp recovered as lnsp[0, 0]. The stub refuses a mixed-level request, so re-merging the two retrievals fails the suite instead of failing at download time. Its synthetic responses use the valid_time and model_level coordinate names the current CDS returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rebase onto dbekaert#808 left _get_from_cds still issuing one merged request for lnsp/z/t/q, which cannot work: CDS returns a mixed-level request as separate files, and where a single file does come back the surface fields are padded onto the full 137-level axis so .squeeze() no longer yields the (lat, lon) arrays calcgeoh and _makeDataCubes expect. _batch_get_from_cds already splits the request correctly and documents why, so rather than repeat that logic, treat a single date as a one-element batch. Both level types now share one download path, which is what stops the two from drifting apart again -- the pl path was already delegating. Also restores the latitude fix from dbekaert#811 in _load_model_level. It was reverted by resolving the rebase conflict in favour of this branch's copy of ecmwf.py, which predates that fix; test_ecmwf_levels.py caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jlmaurer
force-pushed
the
pr/multiday_download
branch
from
August 11, 2026 14:18
d18fddb to
c0fef98
Compare
8 tasks
jlmaurer
marked this pull request as draft
August 11, 2026 14:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
RAiDER access ECMWF ERA-5 data from the ECMWF CDS API. Due to recent changes in how the
era5-reanalysis-completedata is stored (tape archive) data access for model levels is taking a very long for single api calls, on the order of hours. Combine that with two required calls per date and a multi-date request, and you're waiting for days.Description
This PR addresses the issue by bundling the request for all the days together when requesting model levels. The complexity is that pressure level data is still stored on a harddrive, so access is faster, but the multi-day download API does not work the same way. So, this PR updates both the pressure level and model level requests to use the fastest possible data access method.
This PR builds on #795, that PR should be merged first.
How Has This Been Tested?
Unit tests pass locally, and more unit tests have been added to test the new functionality.
Type of change
Checklist: