Add automatic datatime column as needed when running raiderCombine.py - #809
Open
jlmaurer wants to merge 4 commits into
Open
Add automatic datatime column as needed when running raiderCombine.py #809jlmaurer wants to merge 4 commits into
jlmaurer wants to merge 4 commits into
Conversation
jlmaurer
force-pushed
the
pr/gnss_datetime
branch
from
August 11, 2026 14:57
ce748b3 to
4868d89
Compare
jlmaurer
marked this pull request as draft
August 11, 2026 14:58
jlmaurer
marked this pull request as ready for review
August 11, 2026 17:38
Collaborator
Author
The filename carries a single timestamp, so falling back to it stamps every row in the file with the same value. That is right for a single-epoch delay file and silently collapses the time axis for anything else, so log a warning naming the file, the row count and the timestamp rather than doing it quietly. Also record the full scope of this PR in the CHANGELOG: besides the Datetime column, it backfills Lat/Lon/Hgt_m into the GNSS frame from the RAiDER delay file, and aborts with an explanatory error when the two inputs share no common observations instead of failing later on a column that was never created. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jlmaurer
force-pushed
the
pr/gnss_datetime
branch
from
August 11, 2026 18:49
4868d89 to
af71640
Compare
Coveralls flagged this PR's new lines as uncovered: readZTDFile had no tests at all, and neither did the coordinate backfill or the zero-overlap guard in main(). Writing them turned up a bug in readZTDFile that predates this PR. When a file has a Date column but no times column, pd.to_timedelta(0, unit='s') returns a scalar Timedelta, which has no .values for the sum on the next line, so the path always raised AttributeError. The old except (KeyError, ValueError) did not catch that either, so it was never masked -- just never exercised. Keep sec a Series in both branches so to_timedelta returns an array-backed result either way. The new tests cover every branch of readZTDFile: Date with and without times, junk times values coerced to midnight, an existing Datetime column, the filename fallback and its warning, the error when no time information exists at all, and the col_name rename. Two more drive main() end to end on synthetic delay files to check that a GNSS file lacking station coordinates inherits them from the RAiDER file, and that non-overlapping inputs fail with the explanatory error rather than a KeyError further downstream. Module coverage over the GNSS suite goes from 18% to 57%, with every line this PR adds now exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Running
raiderDownloadGNSS.pyand thenraiderCombine.pydirectly fails because the datetime column may not exist. This PR allows for adding it on-the-fly if needed.readZTDFilenow builds theDatetimecolumn from whichever time information the file actually provides, in this order:Datecolumn, optionally combined with atimesseconds-of-day column,Datetimecolumn,YYYYMMDDTHHMMSStimestamp parsed from the filename.When none of those are available it raises an error naming the columns it did find, instead of surfacing a cryptic pandas parsing failure. The filename fallback stamps every row with a single timestamp, which is correct for a single-epoch delay file and lossy for anything covering several epochs, so it logs a warning saying so.
Two further behavior changes worth reviewing deliberately
These go beyond the datetime column and are easy to miss in the diff:
Lat/Lon/Hgt_mare copied into the GNSS frame from the RAiDER delay file, keyed on station ID, for GNSS ZTD files that carry no station coordinates (e.g. UNR per-station delay files). IDs absent from the RAiDER file get NaNs and are dropped downstream, since they could not have been matched anyway.KeyErroron a column that was never created (e.g.sigma_model_neg). This turns a previously-completing — though meaningless — run into an explicit abort, so it is a behavior change for anyone whose inputs do not overlap.Type of change
Checklist: