Create more test bucket to debug high cpu issues

[xueqzhan]

/hold

Summary by CodeRabbit

• New Features
  • Expanded test suite execution to recognize and run additional test groups.
  • Added support for more detailed test counting and repetition when running multiple iterations.
  • Improved execution order by scheduling the new test groups separately during the run.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[xueqzhan]

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

[coderabbitai]

Walkthrough

This PR extends the test suite runner in cmd_runsuite.go to identify and separately handle four new test buckets—DRA, InPlace Resize, Probing, and DeploymentConfig—by adding name-based splits, discovery logging, count duplication logic, and dedicated execution scheduling with monitor interval tracking.

Changes

Test bucket categorization and execution

Layer / File(s)	Summary
New bucket splitting and discovery logging pkg/test/ginkgo/cmd_runsuite.go	Splits kubeTests into draTests, inPlaceResizeTests, probingTests, and openshiftTests into deploymentConfigTests, based on test name matches, and logs discovered counts for each new bucket.
Count duplication bookkeeping pkg/test/ginkgo/cmd_runsuite.go	Saves original slices for the new buckets and extends the --count duplication loop and expectedTestCount calculation to account for them.
Execution scheduling for new buckets pkg/test/ginkgo/cmd_runsuite.go	Schedules and executes the DRA, InPlaceResize, Probing, and DeploymentConfig buckets with copyTests and q.Execute, recording monitor intervals and appending results to the aggregated tests slice.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main change: adding new test buckets to help debug high CPU issues.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The change only adds static test-bucket partitioning/log labels; no Ginkgo It/Describe/Context/When titles or dynamic test names were introduced.
Test Structure And Quality	✅ Passed	Not applicable: the PR only changes the suite runner in cmd_runsuite.go, with no It/BeforeEach/AfterEach/Eventually test-spec code to review.
Microshift Test Compatibility	✅ Passed	No new Ginkgo test definitions were added; the PR only changes test bucketing/execution in cmd_runsuite.go, with no MicroShift-specific APIs introduced.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR only changes the Ginkgo runner/bucketing in cmd_runsuite.go; no new It/Describe/Context tests were added, so the SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	Only test-bucket orchestration in cmd_runsuite.go changed; no manifests/controllers, node selectors, anti-affinity, or replica logic were added.
Ote Binary Stdout Contract	✅ Passed	No stdout writes were added in main/init/test setup; the new bucket logic only logs via logrus and executes tests internally.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo specs were added; the patch only adjusts run-suite bucket scheduling in cmd_runsuite.go.
No-Weak-Crypto	✅ Passed	The only touched code is test-bucketing in cmd_runsuite.go; no MD5/SHA1/DES/RC4/3DES/Blowfish, ECB, custom crypto, or secret/token compares appear.
Container-Privileges	✅ Passed	Only pkg/test/ginkgo/cmd_runsuite.go changed; no manifest or pod security settings were added or modified.
No-Sensitive-Data-In-Logs	✅ Passed	New/changed logs only report fixed bucket names, counts, and durations; no passwords, tokens, PII, hostnames, or customer data are logged.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0c146d90-7bd2-11f1-8628-d6f865f51c18-0

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xueqzhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [xueqzhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

🧹 Nitpick comments (1)

pkg/test/ginkgo/cmd_runsuite.go (1)

691-786: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Repeated bucket-execution boilerplate could be extracted into a helper.

The copyTests → recordTestBucketInterval → q.Execute → EndInterval → log → append sequence is now duplicated ~14 times in this function (4 new instances added here). Extracting a small helper would reduce duplication and make future bucket additions less error-prone.

♻️ Example helper extraction

func executeTestBucket(ctx context.Context, q *parallelTestQueue, recorder monitor.Recorder, name string, tests []*testCase, parallelism int, cfg testOutputConfig, abortFn abortFunc) []*testCase {
	testsCopy := copyTests(tests)
	intervalID, startTime := recordTestBucketInterval(recorder, name)
	q.Execute(ctx, testsCopy, parallelism, cfg, abortFn)
	recorder.EndInterval(intervalID, time.Now())
	logrus.Infof("Completed %s test bucket in %v", name, time.Since(startTime))
	return testsCopy
}

Usage:

-		draTestsCopy := copyTests(draTests)
-		draIntervalID, draStartTime := recordTestBucketInterval(monitorEventRecorder, "DRA")
-		q.Execute(testCtx, draTestsCopy, parallelism, testOutputConfig, abortFn)
-		monitorEventRecorder.EndInterval(draIntervalID, time.Now())
-		logrus.Infof("Completed DRA test bucket in %v", time.Since(draStartTime))
-		tests = append(tests, draTestsCopy...)
+		tests = append(tests, executeTestBucket(testCtx, q, monitorEventRecorder, "DRA", draTests, parallelism, testOutputConfig, abortFn)...)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/test/ginkgo/cmd_runsuite.go` around lines 691 - 786, The bucket execution
flow in this function is repeated many times and should be extracted into a
shared helper. Move the copyTests, recordTestBucketInterval, q.Execute,
monitorEventRecorder.EndInterval, logrus.Infof, and append sequence into a
reusable helper like executeTestBucket, then replace each bucket block (for
example Kubernetes, DRA, InPlaceResize, Probing, Storage, Network, Netpol,
Builds, OpenShift, DeploymentConfig, MustGather) with calls to it. Keep the
helper responsible for returning the copied tests so the caller can continue
appending them to tests.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/test/ginkgo/cmd_runsuite.go`:
- Around line 691-786: The bucket execution flow in this function is repeated
many times and should be extracted into a shared helper. Move the copyTests,
recordTestBucketInterval, q.Execute, monitorEventRecorder.EndInterval,
logrus.Infof, and append sequence into a reusable helper like executeTestBucket,
then replace each bucket block (for example Kubernetes, DRA, InPlaceResize,
Probing, Storage, Network, Netpol, Builds, OpenShift, DeploymentConfig,
MustGather) with calls to it. Keep the helper responsible for returning the
copied tests so the caller can continue appending them to tests.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 62231f2a-4f19-4df7-9419-d5e1cf9a1f93

📥 Commits

Reviewing files that changed from the base of the PR and between a40276a and d398e5b.

📒 Files selected for processing (1)

• pkg/test/ginkgo/cmd_runsuite.go

[xueqzhan]

/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 10

[openshift-ci]

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a8f9bc00-7bd7-11f1-9f83-ccfaed31241a-0

[openshift-merge-bot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[openshift-ci]

@xueqzhan: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-fips	d398e5b	link	true	/test e2e-aws-ovn-fips
ci/prow/e2e-vsphere-ovn-upi	d398e5b	link	true	/test e2e-vsphere-ovn-upi
ci/prow/e2e-vsphere-ovn	d398e5b	link	true	/test e2e-vsphere-ovn
ci/prow/e2e-gcp-ovn	d398e5b	link	true	/test e2e-gcp-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.