Skip to content

Add automatic datatime column as needed when running raiderCombine.py - #809

Open
jlmaurer wants to merge 4 commits into
dbekaert:devfrom
jlmaurer:pr/gnss_datetime
Open

Add automatic datatime column as needed when running raiderCombine.py #809
jlmaurer wants to merge 4 commits into
dbekaert:devfrom
jlmaurer:pr/gnss_datetime

Conversation

@jlmaurer

@jlmaurer jlmaurer commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Description

Running raiderDownloadGNSS.py and then raiderCombine.py directly fails because the datetime column may not exist. This PR allows for adding it on-the-fly if needed.

readZTDFile now builds the Datetime column from whichever time information the file actually provides, in this order:

  1. a Date column, optionally combined with a times seconds-of-day column,
  2. an existing Datetime column,
  3. a YYYYMMDDTHHMMSS timestamp parsed from the filename.

When none of those are available it raises an error naming the columns it did find, instead of surfacing a cryptic pandas parsing failure. The filename fallback stamps every row with a single timestamp, which is correct for a single-epoch delay file and lossy for anything covering several epochs, so it logs a warning saying so.

Two further behavior changes worth reviewing deliberately

These go beyond the datetime column and are easy to miss in the diff:

  • Coordinate backfill. Lat/Lon/Hgt_m are copied into the GNSS frame from the RAiDER delay file, keyed on station ID, for GNSS ZTD files that carry no station coordinates (e.g. UNR per-station delay files). IDs absent from the RAiDER file get NaNs and are dropped downstream, since they could not have been matched anyway.
  • Hard failure on zero overlap. The workflow now raises when the RAiDER and GNSS files share no common observations. Previously the empty frames propagated into the per-station variance analysis and surfaced much later as a KeyError on a column that was never created (e.g. sigma_model_neg). This turns a previously-completing — though meaningless — run into an explicit abort, so it is a behavior change for anyone whose inputs do not overlap.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

  • I have added an explanation of what your changes do and why you'd like us to include them.
  • I have written new tests for your core changes, as applicable.
  • I have successfully ran tests with your changes locally.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jlmaurer jlmaurer changed the title Add automatic datatime column as needed when running raiderCombineGNSS.py Add automatic datatime column as needed when running raiderCombine.py Aug 5, 2026
@jlmaurer
jlmaurer requested a review from sssangha August 5, 2026 12:58
@jlmaurer
jlmaurer marked this pull request as draft August 11, 2026 14:58
@jlmaurer
jlmaurer marked this pull request as ready for review August 11, 2026 17:38
@jlmaurer

Copy link
Copy Markdown
Collaborator Author

@sssangha this one is also ready; note that this is built on top of #808, so that one should be merged first.

jlmaurer and others added 3 commits August 11, 2026 13:20
The filename carries a single timestamp, so falling back to it stamps every row
in the file with the same value. That is right for a single-epoch delay file and
silently collapses the time axis for anything else, so log a warning naming the
file, the row count and the timestamp rather than doing it quietly.

Also record the full scope of this PR in the CHANGELOG: besides the Datetime
column, it backfills Lat/Lon/Hgt_m into the GNSS frame from the RAiDER delay
file, and aborts with an explanatory error when the two inputs share no common
observations instead of failing later on a column that was never created.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coveralls flagged this PR's new lines as uncovered: readZTDFile had no tests at
all, and neither did the coordinate backfill or the zero-overlap guard in main().

Writing them turned up a bug in readZTDFile that predates this PR. When a file
has a Date column but no times column, pd.to_timedelta(0, unit='s') returns a
scalar Timedelta, which has no .values for the sum on the next line, so the path
always raised AttributeError. The old except (KeyError, ValueError) did not catch
that either, so it was never masked -- just never exercised. Keep sec a Series in
both branches so to_timedelta returns an array-backed result either way.

The new tests cover every branch of readZTDFile: Date with and without times,
junk times values coerced to midnight, an existing Datetime column, the filename
fallback and its warning, the error when no time information exists at all, and
the col_name rename. Two more drive main() end to end on synthetic delay files to
check that a GNSS file lacking station coordinates inherits them from the RAiDER
file, and that non-overlapping inputs fail with the explanatory error rather than
a KeyError further downstream.

Module coverage over the GNSS suite goes from 18% to 57%, with every line this PR
adds now exercised.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant