fix(core): dynamically set msg_receive buffer size to prevent CPU spin - #9436
Open
alifakbxr wants to merge 1 commit into
Open
fix(core): dynamically set msg_receive buffer size to prevent CPU spin#9436alifakbxr wants to merge 1 commit into
alifakbxr wants to merge 1 commit into
Conversation
Resolves an issue where a hardcoded 8192-byte `msg_receive()` buffer causes a CPU-spin poison message when `kernel.msgmax` is raised above the default.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Contributor
|
Thanks for submitting the PR! Please complete the CLA as mentioned in the comment above. |
Author
|
@cy-yun Thanks! I have already signed the CLA. @googlebot I signed it! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Top-level Environment Details
Core\Batch(google-cloud-core)BatchJob.phpSummary of Changes & Rationale
This PR resolves a CPU-spin poison message bug that occurs in the background
BatchJobdaemon worker. The worker originally hardcoded a maximum read buffer of8192bytes for the SysV IPC queue. If thekernel.msgmaxis increased and a message larger than 8192 bytes is submitted,msg_receive()would constantly fail with anE2BIGerror, yet never remove the message from the queue, causing an infinite tight loop that maxes out the CPU.This PR replaces the hardcoded
8192byte limit with a dynamic check of the actual SysV queue's max capacity (msg_qbytes) viamsg_stat_queue(), guaranteeing that the read buffer can accommodate the maximum possible message size configured by the OS.Root Cause Analysis
In
BatchJob::run(), the consumer usesmsg_receiveto read items from a SysV IPC queue. It hardcodes8192as the maximum message size. On systems where the defaultkernel.msgmaxis modified to a value larger than 8192, producers (SysvProcessor::submit()) are able to successfully push items larger than 8192 bytes directly onto the queue.When the worker encounters an oversized message,
msg_receive()fails, but because theMSG_NOERRORflag is not passed, the POSIX spec semantics dictate that the oversized message remains in the queue. The worker then immediately loops back around and callsmsg_receive()again without blocking, creating a tight CPU spin.Solution Technical Details
The fix queries the queue's properties using
msg_stat_queue($q). It reads themsg_qbytesvalue, which defines the maximum total size of the queue (which strictly upper-bounds any single message). This value is used as the max message buffer size inmsg_receive().Core/src/Batch/BatchJob.php: Readsmsg_qbytesinto$maxSizeoutside thewhile(true)loop.Core/tests/Unit/Batch/SysvProcessorTest.php: Updated to use the same logic in thereceive()test helper.Testing & Validation Summary
vendor/bin/phpunit --group corepasses perfectly (40 assertions).Reviewer Checklist & Migration Notes
No migration notes required, this is a non-breaking bug fix.