Skip to content

WSA Implementation - #22

Open
mudhoney wants to merge 27 commits into
mainfrom
feature/wsa
Open

WSA Implementation#22
mudhoney wants to merge 27 commits into
mainfrom
feature/wsa

Conversation

@mudhoney

Copy link
Copy Markdown
Contributor

Adds WSA ingestion, a native-HPC coordinate snapshot on every event, and a canonical
footprint shape — plus the tooling to purge and re-collect a single source.

WSA

  • Collects coronal-hole boundaries and magnetic-connectivity footpoints from the CCMC WSA
    dashboard.
  • Registered as source_id 3 and collected by createStandard, chunked one day at a time like
    HEK.
  • Coronal holes are grouped per forecast window, so one event carries all of that window's
    contours.
  • Uses its own HTTP client because CCMC answers 403 to the default user agent.

Coordinates

  • Every event now stores a native-HPC snapshot (x_hpc, y_hpc, footprint_hpc) in arcsec at
    its own coordinate_time.
  • Snapshot resolution moved out of the rotator into HPCResolver, with one strategy per
    coordinate system.
  • Coordinator failover split into FailoverCoordinator, leaving CoordinateRotator to rotate
    only.
  • Added Carrington→helioprojective batch conversion, since WSA footprints are stored in
    degrees and must transform per vertex.
  • The coordinator client now throws if it returns a different number of coordinates than
    were sent.
  • make backfill-hpc fills the snapshot on existing rows, resumable and chunked.
  • Raised the HTTP client timeout to 180s for slow HEK transfers.

Footprints — breaking

  • footprint is now always a list of polygons [[{x,y},…],…], for every source including
    single-polygon ones.
  • Existing rows are converted by the wrap_footprints_in_polygon_list migration.

API

  • frames_with_selections serves the arcsec snapshot as the base and rotates centers only, so
    one delta is valid for every coordinate system.
  • Added ?withDelta to that endpoint for dx/dy offsets instead of absolute coordinates.
  • POST /helioviewer/events/from/{from}/to/{to} accepts path>>uuid selectors to pick a single
    event by id.
  • Removed POST /helioviewer/events/{sources}/observations, superseded by
    frames_with_selections.
  • WSA events no longer report event_type: UNK on the timeline and now carry their API URLs
    in views.
  • WSA is included in the stats endpoint.

Tooling

  • make purge-path PATHS=… deletes an event path and everything attached to it: sidecars,
    rows, region links, distribution buckets, orphaned regions and failure records.
  • make collect SOURCES="A,B" restricts a run to named sources, and make sources lists every
    source with the path it writes to.
  • The collect log marks each registered source [FETCH] or [SKIP], so a filter's effect is
    visible before the run starts.
  • make reset is now a dry run reporting what it would drop, unless APPLY=1.
  • The RHESSI seeder maintains distribution buckets, which it previously ignored entirely.
  • storage/ is gitignored to keep git responsive.

Breaking changes

  • footprint shape. Consumers iterating points directly will now get arrays; they must handle
    the nested shape before this merges.
  • Removed endpoint. Anything still calling POST /helioviewer/events/{sources}/observations
    gets a 404.

Deploy

  • Two migrations run: add_hpc_fields_to_events_table and wrap_footprints_in_polygon_list.
  • Stop the scheduler before migrating — a collection racing the migration writes the old
    footprint shape back.
  • Roll back by restoring the backup, not migrate-rollback: down() cannot unwrap
    multi-polygon footprints, and WSA coronal holes have many.
  • WSA now runs in every scheduled collection, adding roughly 3 minutes to each 6-minute
    cycle and new sustained load on CCMC.
  • Run make backfill-hpc APPLY=1 afterwards; until it completes, endpoints fall back to
    stored values.

mudhoney added 27 commits June 5, 2026 04:39
  Superseded by /helioviewer/events/frames_with_selections (path-selection
  filtering); the only client (helioviewer-api getEventsBatch) has removed
  it and access logs show no traffic. Drops the handler and its private
  getSourceIdByName helper; README/docs notes folded into the
  frames_with_selections sections.
  collect and a make sources listing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant