Skip to content

Add prometheus-adapter rule generation for nrtsearch GPU autoscaling - #4335

Open
vim345 wants to merge 5 commits into
masterfrom
u/mohm/RP-16122
Open

Add prometheus-adapter rule generation for nrtsearch GPU autoscaling#4335
vim345 wants to merge 5 commits into
masterfrom
u/mohm/RP-16122

Conversation

@vim345

@vim345 vim345 commented Jun 16, 2026

Copy link
Copy Markdown

Summary

Teaches setup_prometheus_adapter_config.py to read nrtsearch CRD manifests and generate prometheus-adapter rules for GPU-based autoscaling.

Context

The nrtsearch-operator manages its own HPA (not through PaaSTA's autoscaling framework), so the existing get_rules_for_service_instance path doesn't apply. However, the operator's HPA still needs the prometheus-adapter to serve its custom metric.

This adds a new function get_rules_for_nrtsearch_gpu() that:

  1. Reads {soa_dir}/nrtsearch/nrtsearchserviceeks-{cluster}.yaml
  2. Finds any serverSet with targetGpuUtilization > 0
  3. Generates an adapter rule that joins DCGM_FI_DEV_GPU_UTIL with kube_pod_labels to associate GPU utilization with nrtsearch pods
  4. Divides by the target so the HPA sees 1.0 when at target utilization

The generated metric name follows the existing convention: {deployment-name}-arbitrary_promql-prom.

Test plan

  • New unit tests: rule generation, missing file handling, no-gpu-config skip

@vim345
vim345 requested a review from a team as a code owner June 16, 2026 22:40
@vim345
vim345 force-pushed the u/mohm/RP-16122 branch from f598b62 to ad86f46 Compare June 16, 2026 22:52
@vim345
vim345 requested a review from nemacysts June 16, 2026 22:59
@vim345
vim345 force-pushed the u/mohm/RP-16122 branch from ad86f46 to e32d283 Compare June 18, 2026 21:25
Comment on lines +1111 to +1112
# Generate rules for nrtsearch GPU autoscaling from CRD manifests
rules.extend(get_rules_for_nrtsearch_gpu(paasta_cluster, soa_dir))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we can probably do this without having to add a special case function for nrtsearch

we'd want to update K8S_INSTANCE_TYPE_CLASSES so that

for instance_type_class in K8S_INSTANCE_TYPE_CLASSES:
covers nrtsearch configs

then, we probably need to figure out some typing shenanigans for instance_config in get_rules_for_service_instance(), but then as long as we implement/have implemented all the instance_config.FOO() methods we're calling here, we might not need to do much else re: making this work for nrtsearch?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

especially since serverSets should only have 2 items and only the non-primary is autoscalable, right? ('cause then we can essentially have any of the methods that look up autoscaling data just do that for the non-primary :D)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When series_query is provided in the MetricsProviderDict, the create_instance_arbitrary_promql_scaling_rule path skips the label_replace() wrapping and uses the raw metrics_query/series_query directly. Is that sufficient for the prometheus-adapter to correctly associate the metric with the target deployment, or do we still need the label_replace for deployment/namespace labels in the metrics_query?

is_primary = server_set.get("primary", False)
instance_name = "primary" if is_primary else "replica"
deployment_name = f"{cluster_name}-{instance_name}-dep"
namespace = "paastasvc-nrtsearch"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't these going to be in paasta-nrtsearchservices?

@vim345
vim345 force-pushed the u/mohm/RP-16122 branch from e32d283 to 5a1a071 Compare June 29, 2026 22:35
@vim345
vim345 force-pushed the u/mohm/RP-16122 branch from 5a1a071 to 1695cf8 Compare June 29, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants