Fix(PW): avoid GPU crash in LDOS output - #7786
Open
CleverPhysician wants to merge 1 commit into
Open
Conversation
mohanchen
requested review from
chengleizheng and
mohanchen
and removed request for
chengleizheng
August 8, 2026 07:07
Use a temporary CPU-double PW basis for post-SCF LDOS so synchronized host wavefunctions never enter CPU FFT routines through a GPU-configured basis. Keep the PW LDOS interface non-templated and enable the existing strict GPU regression case.
CleverPhysician
force-pushed
the
fix/pw-gpu-ldos-fft-dispatch
branch
from
August 11, 2026 17:06
844f9cb to
83f379f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the PW-GPU segmentation fault during post-SCF LDOS output by running this short output-only step with a CPU double-precision PW FFT basis. The LDOS API remains non-template, and the existing
scf_out_ldoscase is enabled in the GPU integration test list.Root cause
The LDOS path already consumed the synchronized double-precision host wavefunction, but it invoked the CPU
PW_Basis_K::recip2realpath through a CUDA-configuredPW_Basis_K. The CUDA FFT backend does not provide the CPU staged-FFT virtual methods used by that path, so the call reached a null virtual dispatch after SCF convergence.This explains why the GPU SCF loop itself completed normally and the process failed only when
out_ldosentered post-SCF processing.Changes
cal_ldos_pwas an ordinary, non-template function.pw::setup_pwwfchelper and RAII ownership.K_Vectorsbefore setup so the post-processing basis initialization cannot mutate the calculation's original k-point state.recip2real, LDOS accumulation, MPI/output, and cube-writing logic.tests/01_PW/scf_out_ldosGPU case.pw_seed 1,scf_thr 1e-12, and precision-8 cube output; regenerate the CPU reference without changing the integration comparator's original1e-8tolerance.Design choice and performance
Two implementations were compared on the same RTX 4090 node: (A) the final CPU post-processing basis and (B) a private experimental GPU FFT implementation. The measured end-to-end
out_ldos=1wall times were 1.266944 s and 1.266870 s respectively, a difference of 0.000074 s and well within run-to-run noise. The isolated LDOS increments were approximately 0.036 s and 0.030 s.Because this operation runs once after SCF and the GPU version showed no meaningful end-to-end acceleration for the integration case, the non-template CPU implementation was selected for smaller scope and easier maintenance.
Verification
Environment: one MPI rank,
OMP_NUM_THREADS=1, NVIDIA RTX 4090, CUDA 11.8, GCC 11.5, MPICH 4.3.2.PW_Basis_K::recip2real, called during PW LDOS output.abacus_pw_parabuild: exit code 0.abacus_pw_gpubuild: exit code 0.214573:COMPLETED, exit code0:0.scf_out_ldos: exit code 0; cube comparison passed.scf_out_ldos: exit code 0; cube comparison passed; the cube contained finite, nonzero data.2.000000054e-9, below the unchanged1e-8threshold.out_ldos=0negative control: exit code 0 and no LDOS cube generated.Compute Sanitizer was used on the earlier GPU-FFT candidate, but was not rerun after selecting the final CPU-only LDOS design; it is therefore not claimed as final-patch verification.
Behavior change
PW-GPU calculations requesting post-SCF LDOS output now use a compatible CPU FFT basis for that output step and generate the requested LDOS cube instead of crashing. The SCF loop and
out_ldos=0path are unchanged.The extra CPU basis exists only during post-SCF LDOS output. It does not add work to each SCF iteration, but it introduces temporary host memory proportional to the PW FFT basis when LDOS is requested from a non-CPU/double calculation.
INPUT/documentation impact
No user-facing INPUT keyword, default, or parameter semantics change. Only the integration case explicitly selects a deterministic PW seed, tighter SCF convergence, and higher serialized cube precision.
Core-module impact
The production change is confined to the post-SCF PW LDOS output boundary and reuses existing PW setup, host wavefunction, FFT, and RAII abstractions. It does not introduce a new public abstraction or global dependency.
Limitations
Runtime validation covers
nspin=1, double precision, one MPI rank, one CUDA GPU, and the existingstm_bias=2case. Multi-rank GPU LDOS, single-precision GPU input, other spin modes, other bias values, ROCm, Wannier90, and SOC paths were not runtime-tested.Fixes part of #7770