Skip to content

[CUDA] Avoid copying transposed inputs in row reductions - #4081

Open
Magnushst wants to merge 1 commit into
ml-explore:mainfrom
Magnushst:cuda-reduce-transposed-copy
Open

[CUDA] Avoid copying transposed inputs in row reductions#4081
Magnushst wants to merge 1 commit into
ml-explore:mainfrom
Magnushst:cuda-reduce-transposed-copy

Conversation

@Magnushst

Copy link
Copy Markdown

Proposed changes

CUDA reductions currently materialise every non-contiguous input as a contiguous temporary, even when GeneralContiguousReduce can compute the non-reduction-axis offsets directly.

This change uses the direct row-reduction path for GeneralContiguousReduce inputs when the output contains at least 1,024 elements. General reductions, broadcasted inputs, and smaller outputs retain the existing copy path. This removes one full-input allocation, copy, and kernel launch for sufficiently parallel transposed reductions.

A C++ test checks exact CPU/GPU equivalence for the transposed layout handled by the new path.

Validation

  • CUDA C++ suite: 262 test cases and 3,412 assertions passed.
  • Focused CUDA reduction test: 1 test case and 11 assertions passed.
  • Python reduction suite: 10 tests passed.
  • pre-commit run --all-files passed.
  • git diff --check passed.

The broader Python CUDA suite did not complete because CUDA 13 runtime JIT compilation could not locate cute/tensor.hpp. This header-discovery problem is tracked by #3995; the dedicated reduction suite and complete C++ CUDA suite passed.

Apple Silicon wasn't tested here.

Performance

Environment:

  • NVIDIA GeForce RTX 4070 Laptop GPU, compute capability 8.9
  • Debian under WSL2, kernel 5.15.167.4
  • CUDA 13.2.86
  • GCC 14.2.0
  • CMake 3.31.6
  • Release build
  • Baseline 8d666298652fdac2e7727ecdcf507b1d199bba16

The benchmark interleaved the previous copy-plus-reduce path with the direct path in one warmed process. It used 1,000 warm-up evaluations per path, followed by 15 alternating rounds of 500 evaluations. Reported values are medians of the per-round means.

Transposed input Input size Copy + reduce Direct reduce Change
16x4096x64, axis 1 16 MiB 0.171558 ms 0.038077 ms 77.81% faster
32x2048x128, axis 1 32 MiB 0.424505 ms 0.048314 ms 88.62% faster
16x8192x128, axis 1 64 MiB 0.860658 ms 0.307498 ms 64.27% faster
32x8192x128, axis 1 128 MiB 1.765737 ms 0.611335 ms 65.38% faster

No eligible measured case regressed. Outputs below the 1,024-element threshold retain the existing path. No API or documentation changes are required.

Checklist

Put an x in the boxes that apply.

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

@zcbenz zcbenz added the await verification This pull request is non-trivial and requires a human expert to verify its correctness. label Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants