Bounds check RLE runs in DecodeRLE - #1171
Merged
Merged
Conversation
An RLE run is "<char>*<count>", so decoding one needs both a count byte after the '*' and a preceding character to repeat. Neither was checked, and both are stub controlled: - '*' as the final byte read m_data[index + 1] one past the buffer. - '*' before any literal character evaluated result[result.size() - 1] on an empty string, indexing SIZE_MAX. ASan reports a stack-buffer-overflow for this one. Reject both instead, and log, since a run marker with nothing to repeat means the packet is malformed rather than merely unusual. Once #1166 lands this should raise DebugProtocolError like the other malformed-field paths; keeping it to a log here so the memory-safety fix can land independently. Fixes #1170
Member
Author
bdash
reviewed
Aug 13, 2026
BinaryDecode had the same unchecked count byte and empty-result indexing as DecodeRLE, plus an unchecked read for the byte after a '}' escape. Share one helper between both decoders, reject counts outside the printable range, and return an empty RspData instead of partially decoded data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xusheng6
force-pushed
the
test_1170_decoderle_bounds
branch
from
August 13, 2026 21:07
4c65ec3 to
9cbcce3
Compare
bdash
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1170
DecodeRLEdid not bounds check its'*'branch. An RLE run is<char>*<count>, so itneeds a count byte after the
'*'and a character before it to repeat; neither waschecked, and both come from the remote stub.
'*'as the final byte readm_data[index + 1]one past the buffer.'*'before any literal character evaluatedresult[result.size() - 1]on an emptystring, indexing
SIZE_MAX.RLE is a standard RSP feature that gdbserver uses to compress repeated bytes, so this is
reachable from ordinary traffic, not just a hostile stub.
Change
Reject both cases and log, rather than decoding past the end. A run marker with nothing to
repeat means the packet is malformed. Once #1166 lands this should raise
DebugProtocolErrorlike the other malformed-field paths — kept to a log here so thememory-safety fix can land independently of that PR.
Testing
Under ASan, before and after:
"*A"(leading*)"*""AB*"(trailing*)"T05;*A"Valid RLE still expands:
"A*!"->"AAAAA","0* "->"0000", and packets with no'*'are unchanged.Found by fuzzing the GDB adapter against a live Corellium stub through a mutating proxy;
the path hit was
GetModuleList -> GetRemoteFile -> HostFileIO -> ReceiveRspData -> DecodeRLE.