Skip to content

Bug: Invalid Prometheus metrics format #3594

Description

@frangrech

Bug: RPC method names are used unsanitized as Prometheus metric names, breaking scrapes

Summary

When rpc_enable_metrics=true, the RPC metrics middleware registers a timer per JSON-RPC method using the raw method name straight from the request, with no character sanitization before it's written out as a Prometheus metric name. Any method name containing characters outside [a-zA-Z0-9_:] (e.g. a .) produces an invalid metric line that Prometheus's exposition-format parser rejects — and since Prometheus's parser is strict, this single bad line fails the entire scrape, not just that one metric.

Root cause

  • crates/rpc/rpc-middlewares/src/metrics.rs: register_timer_with_group("async_rpc", name.as_str()) uses req.method_name() verbatim.
  • crates/util/metrics/src/report_prometheus.rs: format!("{}_{}", group, name) concatenates group + method name with no validation/escaping before emitting # TYPE <name> counter.

Steps to reproduce

  1. Start a node with:

    metrics_enabled=true
    rpc_enable_metrics=true
    metrics_prometheus_listen_addr="127.0.0.1:9777"
  2. Send any RPC request whose method name contains a . — it doesn't even need to be a real method, since the metrics middleware runs before method dispatch:

    curl -X POST http://<node>:8545 \
      -H "Content-Type: application/json" \
      -d '{"jsonrpc":"2.0","id":1,"method":"totally.made.up","params":[]}'

    The node correctly rejects it:

    {"jsonrpc":"2.0","id":1,"error":{"code":-32601,"message":"Method not found"}}
  3. Scrape the metrics endpoint:

    curl http://127.0.0.1:9777/metrics | grep made.up

Expected behavior

Metric names are sanitized (e.g. invalid characters replaced with _) so the exposition output is always valid Prometheus format, regardless of what method name a client sends.

Actual behavior

Output includes an invalid line such as:

# TYPE async_rpc_totally.made.up_counter_total counter
async_rpc_totally.made.up_counter_total 1

despite the method never existing. Prometheus fails to parse this and errors the whole scrape (e.g. invalid metric type "made.up_counter_total counter"), so no metrics at all are collected from the node until the offending series ages out or the node is restarted.

Since any client-supplied method name, valid or not, is instrumented unconditionally, this also means an unauthenticated caller can grow the metrics registry indefinitely by sending requests with unique junk method names, which is an unbounded-cardinality / memory-growth concern independent of the Prometheus formatting issue.

Suggested fix

Sanitize name/group in report_prometheus.rs (or at registration time in the RPC metrics middleware) by replacing any character outside [a-zA-Z0-9_:] with _ before building the metric name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions