ci: Add support for CUDA 13.4 ARM64 builds for Windows - #26650
ci: Add support for CUDA 13.4 ARM64 builds for Windows#26650shivamkumard-ctrl wants to merge 4 commits into
Conversation
Added an architecture-specific CUDA 13.4 Windows build entry targeting ARM64. Added a CMake configuration to enable ARM64 CUDA cross-compilation from an x64 Windows environment using the x64-hosted CUDA and MSVC toolchain while linking against the ARM64 CUDA import libraries to produce ggml-cuda.dll. Validated the self-hosted Windows x64 workflow, including toolkit acquisition, CMake configuration, ARM64 CUDA cross-compilation, and packaging. Runtime validation was performed separately on a native ARM64 RTX Spark system using TinyLlama 1.1B Q4_K_M to verify the generated binaries. The ARM64 CUDA job builds only the ggml-cuda.dll backend (LLAMA_BUILD_SERVER=OFF). The release consists of two packages: the main ARM64 release package, which combines the existing ARM64 CPU outputs with ggml-cuda.dll, and a separate runtime package containing the required CUDA runtime libraries (cudart64_13.dll, cublas64_13.dll, and cublasLt64_13.dll). The CUDA 13.4 setup uses NVIDIA Developer Preview component archives instead of the GA component downloads used by the existing CUDA setups and will require updates once CUDA 13.4 reaches GA.
github-actions
Bot
added
build
There was a problem hiding this comment.
1. MSVC has issues with NEON intrinsics, and we should use LLVM/clang for arm builds: #8446 Edit: see we do dynamic runtime linking of MSVC-built ggm-cuda.dll + CLANG-built ggml/lllama already, so this does not apply
2. Any build not containing OpenMP will see severe perf regressions, so this PR is effectively gated by #25782 while not gated (we build and redistribute debug libomp on windows in current CI), we should look to get #25782 merged first
ORippler
left a comment
ORippler
left a comment
There was a problem hiding this comment.
Generally looks good to me, left some comments. After deep-diving I see we do dynamic runtime linking of MVSC-built ggml-cuda.dll + CLANG-built ggml/llama already, so please disregard my first fomment above
| required: true | ||
| cuda_arch: | ||
| description: "CUDA target architecture" | ||
| required: false |
There was a problem hiding this comment.
| required: false | |
| required: true |
If we always specify this, we may as well require it.
There was a problem hiding this comment.
This would break the manual CI flow because it uses the same actions, and we haven’t added the architecture-specific entries there yet. [Reference]
A separate PR will add ARM64 support for manual builds and cover this change.
deep dive on current release.yaml invalidated concerns
I think it's OK if we get this PR before the OpenMP stuff? |
yeah these are legal, not functional/perf aspects |
|
@shivamkumard-ctrl Any reason to still be in draft mode? |
- Moves CUDA-specific CMake options into matrix defines. - Keeps the CUB 3DOT2 option only for CUDA 12.4. - Removes runtime argument construction and the unnecessary server option. - Aligns ARM64 CUDA runtime packaging with the existing robocopy approach. - Generalizes the ARM64 release label from CUDA 13.4 to CUDA 13.
ORippler
left a comment
ORippler
left a comment
There was a problem hiding this comment.
Thanks for adding this!
|
Give us some time to review this PR with @CISC One question that arise - should these builds be tagged as "preview" in the Release page to signify that they use a "preview" version of CUDA 13.4? |
Fine by me/I have no stance on this |
|
LGTM, can we have link to successful |
|
@CISC this one completed https://github.com/shivamkumard-ctrl/llama.cpp/actions/runs/31154609762#artifacts (and was used by me to verify native vs. cross-compiled perf on RTX Spark), but was on commit aa8c22e iirc Latest one is https://github.com/shivamkumard-ctrl/llama.cpp/actions/runs/31171067141#artifacts, where related windows-cuda jobs also pass. LMK if you need/meant something else |
Perfect, thanks. |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
@ORippler guess you get the honor. :) |
4 participants
Overview
Additional information
The CUDA 13.4 setup uses NVIDIA Developer Preview component archives instead of the GA component downloads used by the existing CUDA setups and will require updates once CUDA 13.4 reaches GA.
Requirements