Skip to content

fix(core): dynamically set msg_receive buffer size to prevent CPU spin - #9436

Open
alifakbxr wants to merge 1 commit into
googleapis:mainfrom
alifakbxr:fix/9394-core-batch-cpu-spin
Open

fix(core): dynamically set msg_receive buffer size to prevent CPU spin#9436
alifakbxr wants to merge 1 commit into
googleapis:mainfrom
alifakbxr:fix/9394-core-batch-cpu-spin

Conversation

@alifakbxr

Copy link
Copy Markdown

Top-level Environment Details

Summary of Changes & Rationale

This PR resolves a CPU-spin poison message bug that occurs in the background BatchJob daemon worker. The worker originally hardcoded a maximum read buffer of 8192 bytes for the SysV IPC queue. If the kernel.msgmax is increased and a message larger than 8192 bytes is submitted, msg_receive() would constantly fail with an E2BIG error, yet never remove the message from the queue, causing an infinite tight loop that maxes out the CPU.

This PR replaces the hardcoded 8192 byte limit with a dynamic check of the actual SysV queue's max capacity (msg_qbytes) via msg_stat_queue(), guaranteeing that the read buffer can accommodate the maximum possible message size configured by the OS.

Root Cause Analysis

In BatchJob::run(), the consumer uses msg_receive to read items from a SysV IPC queue. It hardcodes 8192 as the maximum message size. On systems where the default kernel.msgmax is modified to a value larger than 8192, producers (SysvProcessor::submit()) are able to successfully push items larger than 8192 bytes directly onto the queue.

When the worker encounters an oversized message, msg_receive() fails, but because the MSG_NOERROR flag is not passed, the POSIX spec semantics dictate that the oversized message remains in the queue. The worker then immediately loops back around and calls msg_receive() again without blocking, creating a tight CPU spin.

Solution Technical Details

The fix queries the queue's properties using msg_stat_queue($q). It reads the msg_qbytes value, which defines the maximum total size of the queue (which strictly upper-bounds any single message). This value is used as the max message buffer size in msg_receive().

  • Core/src/Batch/BatchJob.php: Reads msg_qbytes into $maxSize outside the while(true) loop.
  • Core/tests/Unit/Batch/SysvProcessorTest.php: Updated to use the same logic in the receive() test helper.

Testing & Validation Summary

  • Unit Tests: vendor/bin/phpunit --group core passes perfectly (40 assertions).
  • Manual Verification: Tested submitting a 9000-byte payload into the queue and manually processing it using a custom script, demonstrating that the consumer successfully handles payloads beyond 8192 bytes gracefully.

Reviewer Checklist & Migration Notes

  • Tested unit tests locally
  • Tested manually
  • Does not contain backward-incompatible changes
  • Follows PSR-2 Coding Style

No migration notes required, this is a non-breaking bug fix.

Resolves an issue where a hardcoded 8192-byte `msg_receive()` buffer causes a CPU-spin poison message when `kernel.msgmax` is raised above the default.
@alifakbxr
alifakbxr requested a review from a team as a code owner August 6, 2026 06:05
@google-cla

google-cla Bot commented Aug 6, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@cy-yun

cy-yun commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for submitting the PR! Please complete the CLA as mentioned in the comment above.

@alifakbxr

Copy link
Copy Markdown
Author

@cy-yun Thanks! I have already signed the CLA. @googlebot I signed it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Core/Batch: hardcoded 8192-byte msg_receive() buffer causes a CPU-spin poison message when kernel.msgmax is raised above the default

2 participants