[test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness#720
[test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness#720wenegiemepraise wants to merge 13 commits into
Conversation
Add an integration suite (test/otel/pernode) and Terraform harness (terraform/eks/daemon/otel-pernode) that validate the Target Allocator per-node allocation strategy end to end, plus the zero-step ServiceMonitor/PodMonitor CRD bundling (G1) and TA resilience to missing CRDs (G2). Suite: - per_node_test.go asserts every scraped series is collected by the agent on the scraped pod's own node (target_node == @resource.k8s.node.name) and that the workload spans >= 2 nodes. - crd_bundling_test.go asserts the SM/PM CRDs are served via discovery after a plain chart install (no prometheus-operator prerequisite). - ta_resilience_test.go asserts the TA Deployment is Available with zero container restarts and that it discovers the monitors once the CRDs exist. - resources/workload.yaml: sm-app/pm-app behind a ServiceMonitor and PodMonitor with a target_node relabel, plus a load generator. Harness installs a helm-charts checkout that bundles the CRDs (no separate CRD install step, by design), deploys custom operator and Target Allocator images carrying the per-node + CRD-watch code, forces the per-node strategy on the CR, applies the workload, and runs the suite.
Validated the pernode suite against a live per-node cluster: - Select Target Allocator pods by app.kubernetes.io/name=cloudwatch-agent- target-allocator (the operator labels component as amazon-cloudwatch-agent-target-allocator, so the previous selector matched nothing). - Assert the TA is currently Ready/Running and not in CrashLoopBackOff instead of requiring a lifetime restart count of 0. The readiness/crashloop check still catches a TA that died on a missing CRD, but is portable across reruns on long-lived clusters; the lifetime restart count is now logged informationally (expected 0 only on a freshly provisioned harness cluster).
- Add local_chart_path var to install the chart from a local checkout (skipping the git clone) so unpushed working-tree changes can be exercised. - Raise the helm_release timeout to 900s for the multi-deployment fresh install (operator, agent DaemonSet, Target Allocator, webhook).
|
|
||
| // Informational: on a freshly provisioned cluster (the otel-pernode harness) | ||
| // this should be 0, evidencing the TA never restarted to pick up the CRDs. | ||
| t.Logf("Target Allocator lifetime container restarts: %d (expected 0 on a fresh harness cluster)", totalRestarts(pods)) |
There was a problem hiding this comment.
The harness installs the chart with CRDs already bundled and then rollout restarts the target-allocator, so this never opens the missing CRD window it's meant to test, and the restart zeroes the one signal (lifetime restart count). Could we relabel it a smoke test on a bundled install rather than CRD resilience?
| // ServiceMonitor and PodMonitor CRDs are present and established (served by the | ||
| // API server) with no manual prerequisite. The otel-pernode harness installs the | ||
| // chart onto a cluster that does NOT pre-install these CRDs, so their presence | ||
| // here is attributable to the chart's bundling. |
There was a problem hiding this comment.
crdServed only checks the CRDs are served, not that this chart installed them, so a rerun or a preexisting prometheus-operator passes even when the chart bundles nothing. Could we assert the CRD carries this release's app.kubernetes.io/managed-by=Helm and treat CRD absence as a precondition?
| topologySpreadConstraints: | ||
| - maxSkew: 1 | ||
| topologyKey: kubernetes.io/hostname | ||
| whenUnsatisfiable: ScheduleAnyway |
There was a problem hiding this comment.
This workload only soft prefers spreading (ScheduleAnyway), but TestPerNodeCoverageAcrossNodes needs two target_nodes, so both replicas can land on one node and flake it. Could we force the spread with DoNotSchedule or a kubernetes.io/hostname podAntiAffinity? Same for pm-app at line 94.
|
|
||
| var results []otelmetrics.MetricResult | ||
| var usedMetric string | ||
| for _, m := range perNodeMetrics { |
There was a problem hiding this comment.
The callers break on the first perNodeMetrics, and sm-app/pm-app share metric names, so one path passing greens the test while the other is broken, and queryWorkloadMetric never checks the test context so it can hang. Could we filter on app/job for one series per path and thread a context.Context through?
|
|
||
| variable "k8s_version" { | ||
| type = string | ||
| default = "1.35" |
There was a problem hiding this comment.
A few defaults are footguns: k8s_version 1.35 may be ahead of what EKS offers in a region (so a default apply fails), and the helm_chart_* defaults point at upstream main. Could we pin a GA version and a real chart ref, and drop the dev registry path in operator_image_repo, before this goes public?
| data "external" "clone_helm_chart" { | ||
| count = var.local_chart_path == "" ? 1 : 0 | ||
| program = ["bash", "-c", <<-EOT | ||
| rm -rf ./helm-charts |
There was a problem hiding this comment.
data.external.clone_helm_chart runs rm -rf then git clone on every plan, which is a side effect in a data source and can wipe a local checkout in the cwd. Could we move it to a null_resource with a triggers guard that clones into path.module behind a [ -d ] check instead?
| provisioner "local-exec" { | ||
| command = <<-EOT | ||
| set -e | ||
| sleep 30 |
There was a problem hiding this comment.
null_resource.patch_cr waits on fixed sleeps instead of readiness (same with validator and the workload apply racing CRD creation), so on a slow cluster kubectl patch fails not found before the CR exists. Could we switch to kubectl wait on the CR and Established CRDs plus kubectl rollout status?
| return groundTruth | ||
| } | ||
|
|
||
| func buildGroundTruth() (*k8sGroundTruth, error) { |
There was a problem hiding this comment.
The groundTruth and TestMain helpers here duplicate the standard suite, so any fallback fix has to land twice, and buildGroundTruth's pod List and nodeNames() are dead code. Could we pull the shared helpers into a common package, drop the dead code, and collapse crdServed's redundant not found branch?
TestTargetAllocatorHealthy claimed to verify G2 CRD-install-ordering resilience, but the harness bundles the CRDs at install so the missing-CRD window is never opened, and a rollout restart would zero the lifetime restart signal. Rename to TestTargetAllocatorHealthyOnBundledInstall and reword the assertions/comments to describe what it actually checks (TA healthy on a bundled install). The missing- CRD tolerance itself is covered by the operator's Target Allocator unit tests.
crdServed only checked the CRDs were served, so a rerun or a pre-existing prometheus-operator passed even when the chart bundled nothing. Fetch the CRD objects via a dynamic client and assert app.kubernetes.io/managed-by=Helm and meta.helm.sh/release-namespace == the release namespace, so only CRDs this chart release installed count. Treat CRD absence as a precondition failure. Drop the now-unused discovery-based crdServed helper.
sm-app/pm-app used topologySpreadConstraints with whenUnsatisfiable: ScheduleAnyway (soft), so both replicas could land on one node and flake TestPerNodeCoverageAcrossNodes, which needs two distinct target_nodes. Switch both to DoNotSchedule (hard spread on kubernetes.io/hostname) so the two replicas are guaranteed on separate nodes.
queryWorkloadMetric broke on the first metric and sm-app/pm-app share metric names, so the ServiceMonitor path passing greened the test while a broken PodMonitor path went unnoticed; it also used context.Background() and could hang. Stamp a deterministic app label per monitor (sm-app/pm-app) via relabel, filter queries by app, validate each path independently (subtests), and thread a context.Context with a bounded deadline through queryWorkloadMetric.
Pin k8s_version to a GA EKS version (1.35 -> 1.31) so a default apply does not fail in regions that do not yet offer it; scrub the dev registry path from the operator_image_repo example; and correct the helm chart-source comment (it does not default to a fork) to warn that upstream main lacks the feature until the stack merges and to pin a fixed ref / use local_chart_path.
The helm chart was fetched in a data source that ran rm -rf ./helm-charts then git clone on every plan -- a side effect in a data source that could wipe a checkout in the caller's cwd. Replace with a null_resource that clones into path.module behind a git-dir guard, triggered only when the repo/branch changes (fetch+checkout on change, clone once otherwise). Point chart_path and the helm_release dependency at it.
patch_cr slept 30s hoping the CR existed and the workload apply raced CRD creation. Replace with kubectl wait --for=condition=Established on the AmazonCloudWatchAgent CRD (then poll for the CR) before patching, and on the bundled ServiceMonitor/PodMonitor CRDs before applying the workload. Drop the validator's fixed 3-minute propagation sleep -- the tests poll CloudWatch with a bounded retry. terraform validate passes.
buildGroundTruth listed all pods into a pods map that nothing reads, and nodeNames() was never called. Remove the pods field, the pod List, and nodeNames(); ground truth now only holds nodes (all that the per-node checks use). (crdServed and its redundant not-found branch were already removed when the CRD-bundling test moved to a Helm-ownership check.)
Propagate the aws#720 review fixes (bundled-install smoke test, CRD Helm-ownership assertion, workload DoNotSchedule spread, both-path metric validation + context, terraform footguns/clone/readiness, dead-code removal) so the routing suite builds on the updated harness.
2 participants
Summary
Add the
test/otel/pernodeintegration suite plus a Terraform EKS harness that validatesthe ServiceMonitor/PodMonitor scraping path end to end on a live cluster: per-node target
locality, zero-step CRD bundling, and Target Allocator resilience to missing CRDs.
What's included
Suite (
test/otel/pernode/, build tagintegration):per_node_test.go— every SM/PM series is node-local:target_node == @resource.k8s.node.name; targets span ≥2 nodes.crd_bundling_test.go— SM + PM CRDs are served/established after a plain chart install (harness never installs them separately).ta_resilience_test.go— TA Deployment Available, no CrashLoopBackOff, 0 restarts on a fresh cluster; and it actually discovers monitors.setup_test.go,k8s_helpers_test.go,metrics_test.go— shared clientset/ground-truth, CRD discovery, otelmetrics client bootstrap.resources/workload.yaml— sm-app/pm-app workloads + SM/PM with thetarget_noderelabel + load generator.Harness (
terraform/eks/daemon/otel-pernode/): provisions EKS, installs the chart with theper-node-capable operator/TA images, applies the workloads, and runs the suite. Because it
exercises unreleased code it takes
operator_image_*,ta_image,helm_chart_repo/branchvars.Verification method (per-node)
Dependencies / notes
once those and their custom images are available; tests are skip-guarded so they don't break
the broader suite where fixtures are absent.
go vet -tags integration ./test/otel/pernode/...is clean against currentmain.