feat: add SpatialLMQwenForCausalLM converter - #22
Open
CodeDoes wants to merge 12 commits into
Open
Conversation
Reorder from contiguous write to read with atomic stores.
…ecovery (aifoundry-org#15) This PR adds Q4_K MUL_MAT to the ET (ETSOC-1) backend — a scalar kernel plus a matrix-engine kernel for prefill — improves the existing Q4_0/F16/F32 matrix-engine kernels, and fixes a Q8_0 generation regression in the uberkernel path. Verified on Llama-3.2-1B-Instruct on ETSOC-1 with flash attention enabled.
…sor-engine & vectorized dots, plus uberkernel support (aifoundry-org#16) This PR adds full ET-backend support for the Q2_K, Q3_K, Q5_K, and Q6_K super-block quantizations (previously only Q4_0/Q8_0/Q4_K existed), and substantially improves generation throughput across all K-quants — including the pre-existing Q4_K — by (a) vectorizing the generation dot product and (b) enabling the K-quant matmuls to run inside the uberkernel. For prefill, every K-quant gets a tensor-unit (matrix-engine) kernel delivering very large token/sec. Everything is validated on test-backend-ops (MUL_MAT + GET_ROWS) and by coherent end-to-end generation on Llama-3.2-1B models.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Isolated SpatialLM converter that was originally part of PR #19. The reviewer asked for
conversion/*to be reverted from the kernel PR, so it is split into its own PR.What changed
conversion/__init__.py— add"SpatialLMQwenForCausalLM": "qwen"toTEXT_MODEL_MAPso the converter registry knows which model class to dispatch to.conversion/qwen.py— addSpatialLMQwenModelclass (registered asSpatialLMQwenForCausalLM). It inherits fromQwen2Modelbut strips three groups of 3D-point encoder tensors before writing the GGUF, so the spatial encoder weights are not serialized into the model file:point_backbone.*point_proj.*point_Why it's a separate PR
PR #19 mixed kernel correctness fixes, a performance change, and a new model converter. Marty asked for
conversion/*to be removed from that series; this PR satisfies that without reverting the converter work entirely. If the SpatialLM support is unwanted, close this PR — do not revert it by touching the kernel PR.Review notes
point_*filter is prefix-based; if downstream checkpoints introduce other point-encoder tensors with a different prefix, add them here explicitly.model_arch = gguf.MODEL_ARCH.QWEN2matches the base Qwen2 architecture; if SpatialLM ever diverges enough to need its own arch key, it will need a corresponding GGUF reader update on the llama.cpp side.