Skip to content

fix(workers): guard metricViewConfig before dereferencing queryConfigs in telemetry monitor evaluation - #3112

Open
Nedjagang wants to merge 1 commit into
OneUptime:masterfrom
Nedjagang:fix/monitor-metricviewconfig-nullcheck
Open

fix(workers): guard metricViewConfig before dereferencing queryConfigs in telemetry monitor evaluation#3112
Nedjagang wants to merge 1 commit into
OneUptime:masterfrom
Nedjagang:fix/monitor-metricviewconfig-nullcheck

Conversation

@Nedjagang

Copy link
Copy Markdown
Contributor

Summary

  • Every monitor-type handler in MonitorTelemetryMonitor.ts (Metric, Kubernetes, Docker, Host, Podman, Proxmox, IoT, Docker Swarm, Ceph) null-checks the top-level monitor step config, then immediately dereferences config.metricViewConfig.queryConfigs without checking that metricViewConfig itself is present.
  • A monitor whose step data is missing metricViewConfig (e.g. saved mid-flow before a template/form finished populating it) throws an unhandled TypeError: Cannot read properties of undefined (reading 'queryConfigs') on every evaluation attempt.
  • Because the job throws instead of completing, it gets re-queued indefinitely. In our deployment this kept queue-size-based KEDA autoscaling metrics permanently inflated, driving worker replicas toward the configured max while never actually draining the backlog, since the "backlog" was a job that could never succeed.
  • This adds the same missing-config guard already used for the top-level config to metricViewConfig in all 9 affected handlers, so a malformed monitor throws a clean BadDataException instead of crashing and retrying forever.

Root cause (observed in production)

We hit this with monitor records shaped like { "cephMonitor": { "clusterIdentifier": "..." } } — no resourceFilters, metricViewConfig, or rollingTime — created via a template/onboarding flow that appears to save the step data before the rest of the form is filled in. Every scheduled evaluation of that monitor crashed identically, and since the worker treats a thrown job as "needs retry," it never stopped retrying. This happened simultaneously across all 9 vulnerable monitor types (each project had one [Template] X Monitor per type), producing a sustained, artificial load signal that scaled workers to near their configured ceiling.

Fix

Minimal, mechanical, no behavior change for well-formed monitors — just fails fast with a clear error instead of throwing a raw TypeError for malformed ones.

Test plan

  • npx tsc --noEmit on the App workspace shows no new errors introduced by this change (pre-existing unrelated errors in the workspace are untouched)
  • Would appreciate maintainer guidance on whether there's an existing unit test suite for MonitorTelemetryMonitor.ts this should extend — happy to add a case per handler exercising a monitor step with metricViewConfig unset if pointed at the right test file.

…s in telemetry monitor evaluation

Every monitor-type handler in MonitorTelemetryMonitor.ts (Metric, Kubernetes,
Docker, Host, Podman, Proxmox, IoT, Docker Swarm, Ceph) null-checks the
top-level monitor step config but then dereferences
`config.metricViewConfig.queryConfigs` without checking that
`metricViewConfig` itself is present. A monitor whose step data is missing
metricViewConfig (e.g. saved before the form/template flow finished
populating it) throws an unhandled TypeError on every evaluation attempt.
Since the evaluation job fails instead of completing, it keeps getting
re-queued, which can inflate queue-size-based autoscaling metrics and drive
worker replicas up indefinitely chasing a job that can never succeed.

Add the same missing-config guard already used for the top-level config to
metricViewConfig in each of the 9 handlers, so a malformed monitor throws a
clean BadDataException instead of crashing.
@simlarsen

Copy link
Copy Markdown
Contributor

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants