Skip to content

feat(logs): add Logs pillar with bounded log tails - #64

Merged
thiagoesteves merged 6 commits into
mainfrom
thiagoesteves/logs-pillar
Jul 17, 2026
Merged

feat(logs): add Logs pillar with bounded log tails#64
thiagoesteves merged 6 commits into
mainfrom
thiagoesteves/logs-pillar

Conversation

@thiagoesteves

Copy link
Copy Markdown
Owner

What

Adds a Logs pillar: a bounded, read-only tail of the selected node's log files during an incident - the web equivalent of observer_cli's log tail pane.

  • ObserverWeb.Logs.list_handlers/1 discovers the file-backed :logger handlers on the target node (:logger.get_handler_config/0); console handlers are skipped.
  • ObserverWeb.Logs.tail/3 reads at most the selected tail size (16 KB to 1 MB, hard-capped) from the end of the file via a single stdlib-only RPC - a pre-parsed :erl_eval expression opens/preads/closes on the remote node, so it works against any OTP node regardless of the observer_web version there (same principle as ObserverWeb.SystemInfo).
  • The page offers service, log file and tail size selectors plus REFRESH, and shows file size / truncation chips above the tail pane.

Security notes

  • Only files exposed by the node's logger handler configs can ever be requested; the form's file value is validated against that list on every read (selected_file/2 falls back to the first known handler, and ObserverWeb.Logs.tail/3 re-checks server-side), so free-form paths from the browser are never opened.
  • Reads are bounded and use :raw mode, keeping the descriptor inside the rpc process.

Why

Part of the roadmap derived from comparing ObserverWeb against OTP observer, observer_cli and Phoenix LiveDashboard: observer_cli proved how useful a safe bounded log view is during incidents, and it sits naturally next to the Crashdump pillar.

Risk assessment

  • Impact: new read-only pillar; no changes to existing pages.
  • Blast radius: additive - new context module and page, plus three small wiring edits (page resolution, nav list, icon).
  • Regression risk: low - bounded reads, allowlisted files, existing Rpc adapter; suite green (408 tests, 95.8% coverage), credo/sobelow/dialyzer/format clean. Note: there is a pre-existing intermittent failure in version/tracing tests that reproduces on main; a separate fix PR will follow.
  • Rollback plan: revert the commit; no config or data involved.

Checklist

  • mix test green (408 tests)
  • mix coveralls 95.8% (threshold 95%)
  • mix credo --strict, mix sobelow, mix dialyzer, mix format --check-formatted clean
  • Small focused diff, no leftover debug output

🤖 Generated with Claude Code

Add a new Logs page that reads the last chunk of the selected node's
log files, the web equivalent of observer_cli's log tail pane.

ObserverWeb.Logs restricts sources to the file-backed :logger handlers
configured on the target node - free-form paths are never accepted, so
the dashboard cannot read arbitrary files. The tail itself is a single
stdlib-only RPC: a pre-parsed :erl_eval expression preads at most
max_bytes (capped at 1 MB) from the end of the file on the remote node,
so nothing is required on the target beyond OTP itself. Truncated
chunks drop the partial first line.

Risk assessment:
- Impact: new read-only pillar (nav entry, page, context module); no
  changes to existing pages.
- Blast radius: additive - new files plus three small wiring edits
  (index resolve_page, nav list, icon).
- Regression risk: low - reads are bounded, restricted to logger
  handler files and go through the existing Rpc adapter; suite green
  (408 tests, 95.8% coverage). The single intermittent failure seen
  locally reproduces on main (version/tracing tests) and is unrelated;
  a separate fix will follow.
- Rollback plan: revert this commit; no config or data involved.
When the selected node exposes no file-backed logger handlers, the Log
File select renders empty and the phx-change payload arrives without a
"file" key at all, so the %{map | key} update in the :logs_refresh
handler raised KeyError and crashed the LiveView. Build the form params
with Map.put instead (same for the nodedown service fallback).

Also document how to make logs visible: new "Logs" section in the
installation guide plus a ObserverWeb.Logs moduledoc section with the
Elixir 1.15+ logger file-handler configuration and the runtime
:logger.add_handler alternative.

Risk assessment:
- Impact: fixes a LiveView crash on the new Logs page; no behavior
  change when file handlers exist.
- Blast radius: two form-params call sites in the Logs page plus docs.
- Regression risk: low - regression test reproduces the exact KeyError
  without the fix and passes with it; suite green (409 tests, 95.8%
  coverage).
- Rollback plan: revert the commit.
@thiagoesteves

Copy link
Copy Markdown
Owner Author

Pushed a fix for the crash reported while testing: selecting a service with no file-backed logger handlers rendered an empty Log File select, so the change payload had no "file" key and the %{map | key} update in handle_info(:logs_refresh, ...) raised KeyError, killing the LiveView. Form params are now built with Map.put, with a regression test that reproduces the exact crash without the fix.

Also added the missing documentation: an installation-guide "Logs" section and a ObserverWeb.Logs moduledoc section covering the Elixir 1.15+ file-handler config (config :my_app, :logger, [{:handler, ...}] + Logger.add_handlers/1) and the runtime :logger.add_handler/3 alternative - nothing needs configuring on the Observer Web side, the page follows whatever file handlers the observed node has.

…ogs page

Attach a file-backed :logger handler in dev.exs so the standalone dev
server always has something for the Logs pillar to tail. Each instance
writes to /tmp/observer_web_dev_<node>_<random>.log - node slug plus a
random suffix - so the multi-node workflows from the README (observer +
broadcast side by side) never collide. A heartbeat appends info/warning/
error lines every 5s so REFRESH always shows fresh content.

Toggles: OBSERVER_WEB_DEV_LOG_FILE=false skips the handler,
OBSERVER_WEB_DEV_LOG_HEARTBEAT_MS tunes or silences the heartbeat.

Risk assessment:
- Impact: dev.exs only; the published library is untouched.
- Blast radius: standalone dev server boot path.
- Regression risk: low - verified by booting the dev server and
  tailing the generated /tmp file through the Logs page.
- Rollback plan: revert the commit or set OBSERVER_WEB_DEV_LOG_FILE=false.
@thiagoesteves

Copy link
Copy Markdown
Owner Author

Added a dev.exs setup for local testing: every instance attaches a file-backed logger handler writing to /tmp/observer_web_dev_<node>_<random>.log (random suffix so multiple side-by-side nodes never collide), with a 5s heartbeat producing info/warning/error lines so the tail always has fresh content. OBSERVER_WEB_DEV_LOG_FILE=false disables it, OBSERVER_WEB_DEV_LOG_HEARTBEAT_MS tunes the heartbeat. Verified by booting the dev server and tailing the generated file through the Logs page.

The dev.exs file handler used Logger.Formatter defaults, which enable
ANSI colors when the server runs in a terminal - landing escape codes
([33m, [0m, ...) in the log file. Disable colors for the file
formatter, and defensively strip ANSI sequences in the Logs pane since
any real-world log written by a color-enabled formatter has the same
problem.

Risk assessment:
- Impact: cleaner dev log files; the pane now renders colored logs
  legibly instead of showing raw escape codes.
- Blast radius: dev.exs formatter plus one display-time transform in
  the Logs page; ObserverWeb.Logs still returns raw bytes.
- Regression risk: low - covered by a page test with ANSI content and
  verified against a running dev server (zero escape bytes in file).
- Rollback plan: revert the commit.
@thiagoesteves

Copy link
Copy Markdown
Owner Author

Follow-up from local testing: the dev file handler was writing ANSI color codes ([33m/[0m) into the log because Logger.Formatter enables colors when running in a terminal. The dev.exs formatter now disables colors, and the Logs pane also strips ANSI sequences at display time - real-world log files written by color-enabled formatters have the same issue, so the pane stays legible either way.

Render the tail as parsed log entries instead of a raw pre dump. A new
entry starts at a line that looks like a log head (time, date or an
Erlang report banner); stack traces and wrapped output attach to the
entry above. Each entry renders as a single truncated line with a
triangle marker: filled and clickable when there is more to expand
(multi-line entries, or single lines beyond 160 chars that truncation
clips), hollow and inert when the line is all there is. Multi-line
entries also advertise hidden content with a dimmed ellipsis, and
error/warning summaries are color-coded.

The marker pattern lives in a reusable Core.disclosure component
(summary/body slots plus an expandable? flag) so other tables can adopt
the same visual language later.

The dev.exs heartbeat now exercises all of it: a multi-line fake crash
report every 10th beat and a deliberately long single-line message
every 7th. Also fix a silent filter: config.exs pins the primary
logger level to :warning, which dropped info entries before any
handler saw them - the dev server now lowers the primary level to
:info while pinning the console handler back to :warning, so the
terminal stays as quiet as before while the file gets the full mix.

Risk assessment:
- Impact: Logs page rendering only, plus dev-server-only logger tweaks;
  ObserverWeb.Logs still returns raw bytes.
- Blast radius: Logs page, one new Core component (used only by Logs),
  dev.exs.
- Regression risk: low - entry parsing is display-time only and
  covered by tests for grouping, markers, long lines, ANSI and levels;
  suite green (412 tests, 95.8% coverage), credo/sobelow/dialyzer clean.
- Rollback plan: revert the commit.
@thiagoesteves

Copy link
Copy Markdown
Owner Author

Logs pane UX rework after visual testing:

  • The tail is parsed into log entries (a new entry starts at a time/date/Erlang-report head; stack traces attach to the entry above) and each entry renders as one truncated line.
  • Disclosure rows follow a filled/hollow marker scheme: ▶ filled + clickable when there is more to expand (multi-line entries, or long single lines that truncation clips), ▷ hollow + inert when the line is all there is. Multi-line entries also show a dimmed , and error/warning summaries are color-coded.
  • The pattern is extracted into a reusable Core.disclosure component so other tables (tracing results, ETS previews) can adopt it later.
  • dev.exs heartbeat now emits a multi-line fake crash report and a deliberately long single-line message to exercise the UI, and fixes a silent filter: the primary logger level (:warning from config.exs) dropped info entries before any handler saw them - the dev server lowers it to :info while pinning the console handler back to :warning.

Add a Refresh interval selector (Paused/2s/5s/10s, default 5s) using
the same generation-counted tick chain as the Network pillar: changing
any control or pressing REFRESH restarts the chain and stale in-flight
ticks are ignored. The entries pane reuses the existing ScrollBottom
hook, so it follows the newest entries after each refresh but stays
pinned when the user scrolls away from the bottom.

Risk assessment:
- Impact: Logs page gains periodic refresh; manual REFRESH behavior
  unchanged apart from also resetting the timer cadence.
- Blast radius: Logs page only; the tick pattern and scroll hook are
  reused as-is from Network/tracing.
- Regression risk: low - tick chain covered by an interval test and a
  stale-generation test; suite green (414 tests, 95.9% coverage),
  credo/sobelow/dialyzer/format clean.
- Rollback plan: revert the commit.
@thiagoesteves

Copy link
Copy Markdown
Owner Author

Added auto refresh: a Refresh interval selector (Paused/2s/5s/10s, default 5s) driven by the same generation-counted tick chain the Network pillar uses, and the entries pane now follows the newest entries via the existing ScrollBottom hook - it re-sticks to the bottom after each refresh unless the user has scrolled away, in which case the position stays pinned.

@thiagoesteves
thiagoesteves merged commit 58601af into main Jul 17, 2026
5 of 6 checks passed
@thiagoesteves
thiagoesteves deleted the thiagoesteves/logs-pillar branch July 17, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant