You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
GatherQMM::eval_gpu normalizes w, scales and biases with ensure_row_contiguous_matrix, which validates only the last two axes. A row
slice of an [E, 2R, D] expert weight passes that check while keeping the
leading stride of the original array, so the sorted path reads every expert
after the first from the wrong offset — silently, with no error.
The gather kernels index the leading axes themselves, so this switches those
three inputs to ensure_row_contiguous, which already exists in the file and
is already used for indices. x is left as is; the failure I can demonstrate
is in the weight path.
Verified on M5 Pro / macOS 26.5.1: the new test fails before the change and
passes after, and Ran 794 tests ... OK (skipped=46) for the full suite.
dudududukim
changed the title
Use the full row-contiguity check for GatherQMM's quantized inputs
Normalize biases before encoding in gather_qmm_rhs
Aug 8, 2026
Thank you for pointing that out — it led to the actual root cause.
The biases ensure_row_contiguous runs mid-encode: after set_compute_pipeline_state and the first three set_input_array calls. For
a non-contiguous biases it encodes a copy kernel on the same encoder right
there, clobbering the state of the kernel being set up. w and scales are
normalized at the top of the function, so they were never affected — isolating
the inputs on 0.32.0 confirms only strided biases reproduces the bug
(w only: 0.0, scales only: 0.0, biases only: 1.41).
Updated the PR: the biases normalization is hoisted next to w/scales in
both gather_qmm_rhs and gather_qmm_rhs_nax, and the eval_gpu change is
reverted. Test unchanged, red before / green after, full test_quantized
passes on M5 Pro.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Fixes #4055.
GatherQMM::eval_gpunormalizesw,scalesandbiaseswithensure_row_contiguous_matrix, which validates only the last two axes. A rowslice of an
[E, 2R, D]expert weight passes that check while keeping theleading stride of the original array, so the sorted path reads every expert
after the first from the wrong offset — silently, with no error.
The gather kernels index the leading axes themselves, so this switches those
three inputs to
ensure_row_contiguous, which already exists in the file andis already used for
indices.xis left as is; the failure I can demonstrateis in the weight path.
Verified on M5 Pro / macOS 26.5.1: the new test fails before the change and
passes after, and
Ran 794 tests ... OK (skipped=46)for the full suite.Checklist
Put an
xin the boxes that apply.pre-commit run --all-filesto format my code / installed pre-commit prior to committing changes