Add prometheus-adapter rule generation for nrtsearch GPU autoscaling - #4335
Add prometheus-adapter rule generation for nrtsearch GPU autoscaling#4335vim345 wants to merge 5 commits into
Conversation
| # Generate rules for nrtsearch GPU autoscaling from CRD manifests | ||
| rules.extend(get_rules_for_nrtsearch_gpu(paasta_cluster, soa_dir)) |
There was a problem hiding this comment.
i think we can probably do this without having to add a special case function for nrtsearch
we'd want to update K8S_INSTANCE_TYPE_CLASSES so that
then, we probably need to figure out some typing shenanigans for instance_config in get_rules_for_service_instance(), but then as long as we implement/have implemented all the instance_config.FOO() methods we're calling here, we might not need to do much else re: making this work for nrtsearch?
There was a problem hiding this comment.
especially since serverSets should only have 2 items and only the non-primary is autoscalable, right? ('cause then we can essentially have any of the methods that look up autoscaling data just do that for the non-primary :D)
There was a problem hiding this comment.
When series_query is provided in the MetricsProviderDict, the create_instance_arbitrary_promql_scaling_rule path skips the label_replace() wrapping and uses the raw metrics_query/series_query directly. Is that sufficient for the prometheus-adapter to correctly associate the metric with the target deployment, or do we still need the label_replace for deployment/namespace labels in the metrics_query?
| is_primary = server_set.get("primary", False) | ||
| instance_name = "primary" if is_primary else "replica" | ||
| deployment_name = f"{cluster_name}-{instance_name}-dep" | ||
| namespace = "paastasvc-nrtsearch" |
There was a problem hiding this comment.
aren't these going to be in paasta-nrtsearchservices?
…PE_CLASSES framework
Summary
Teaches
setup_prometheus_adapter_config.pyto read nrtsearch CRD manifests and generate prometheus-adapter rules for GPU-based autoscaling.Context
The nrtsearch-operator manages its own HPA (not through PaaSTA's autoscaling framework), so the existing
get_rules_for_service_instancepath doesn't apply. However, the operator's HPA still needs the prometheus-adapter to serve its custom metric.This adds a new function
get_rules_for_nrtsearch_gpu()that:{soa_dir}/nrtsearch/nrtsearchserviceeks-{cluster}.yamltargetGpuUtilization > 0DCGM_FI_DEV_GPU_UTILwithkube_pod_labelsto associate GPU utilization with nrtsearch podsThe generated metric name follows the existing convention:
{deployment-name}-arbitrary_promql-prom.Test plan