Skip to content

Fix(PW): avoid GPU crash in LDOS output - #7786

Open
CleverPhysician wants to merge 1 commit into
deepmodeling:developfrom
CleverPhysician:fix/pw-gpu-ldos-fft-dispatch
Open

Fix(PW): avoid GPU crash in LDOS output#7786
CleverPhysician wants to merge 1 commit into
deepmodeling:developfrom
CleverPhysician:fix/pw-gpu-ldos-fft-dispatch

Conversation

@CleverPhysician

@CleverPhysician CleverPhysician commented Aug 7, 2026

Copy link
Copy Markdown

Summary

Fix the PW-GPU segmentation fault during post-SCF LDOS output by running this short output-only step with a CPU double-precision PW FFT basis. The LDOS API remains non-template, and the existing scf_out_ldos case is enabled in the GPU integration test list.

Root cause

The LDOS path already consumed the synchronized double-precision host wavefunction, but it invoked the CPU PW_Basis_K::recip2real path through a CUDA-configured PW_Basis_K. The CUDA FFT backend does not provide the CPU staged-FFT virtual methods used by that path, so the call reached a null virtual dispatch after SCF convergence.

This explains why the GPU SCF loop itself completed normally and the process failed only when out_ldos entered post-SCF processing.

Changes

  • Keep cal_ldos_pw as an ordinary, non-template function.
  • Reuse the main PW basis for CPU double-precision calculations.
  • For other backends or precisions, create a narrow-scope CPU double-precision PW basis with the existing pw::setup_pwwfc helper and RAII ownership.
  • Copy K_Vectors before setup so the post-processing basis initialization cannot mutate the calculation's original k-point state.
  • Continue using the synchronized host wavefunction and the existing CPU recip2real, LDOS accumulation, MPI/output, and cube-writing logic.
  • Add no CUDA-specific buffer management, device copy, new global state, third-party dependency, or template propagation to the LDOS path.
  • Enable the existing tests/01_PW/scf_out_ldos GPU case.
  • Stabilize that case with pw_seed 1, scf_thr 1e-12, and precision-8 cube output; regenerate the CPU reference without changing the integration comparator's original 1e-8 tolerance.

Design choice and performance

Two implementations were compared on the same RTX 4090 node: (A) the final CPU post-processing basis and (B) a private experimental GPU FFT implementation. The measured end-to-end out_ldos=1 wall times were 1.266944 s and 1.266870 s respectively, a difference of 0.000074 s and well within run-to-run noise. The isolated LDOS increments were approximately 0.036 s and 0.030 s.

Because this operation runs once after SCF and the GPU version showed no meaningful end-to-end acceleration for the integration case, the non-template CPU implementation was selected for smaller scope and easier maintenance.

Verification

Environment: one MPI rank, OMP_NUM_THREADS=1, NVIDIA RTX 4090, CUDA 11.8, GCC 11.5, MPICH 4.3.2.

  • Reproduced the original GPU failure twice with exit code 139 after eight converged SCF iterations.
  • GDB located the host-side null dispatch in PW_Basis_K::recip2real, called during PW LDOS output.
  • Final CPU-only translation-unit syntax build: exit code 0.
  • Final CUDA translation-unit syntax build: exit code 0.
  • Final CPU abacus_pw_para build: exit code 0.
  • Final CUDA abacus_pw_gpu build: exit code 0.
  • Final Slurm validation job 214573: COMPLETED, exit code 0:0.
  • CPU scf_out_ldos: exit code 0; cube comparison passed.
  • CUDA scf_out_ldos: exit code 0; cube comparison passed; the cube contained finite, nonzero data.
  • CUDA maximum absolute cube difference from the CPU reference: 2.000000054e-9, below the unchanged 1e-8 threshold.
  • CPU cube was byte-identical to the checked-in reference.
  • CUDA out_ldos=0 negative control: exit code 0 and no LDOS cube generated.

Compute Sanitizer was used on the earlier GPU-FFT candidate, but was not rerun after selecting the final CPU-only LDOS design; it is therefore not claimed as final-patch verification.

Behavior change

PW-GPU calculations requesting post-SCF LDOS output now use a compatible CPU FFT basis for that output step and generate the requested LDOS cube instead of crashing. The SCF loop and out_ldos=0 path are unchanged.

The extra CPU basis exists only during post-SCF LDOS output. It does not add work to each SCF iteration, but it introduces temporary host memory proportional to the PW FFT basis when LDOS is requested from a non-CPU/double calculation.

INPUT/documentation impact

No user-facing INPUT keyword, default, or parameter semantics change. Only the integration case explicitly selects a deterministic PW seed, tighter SCF convergence, and higher serialized cube precision.

Core-module impact

The production change is confined to the post-SCF PW LDOS output boundary and reuses existing PW setup, host wavefunction, FFT, and RAII abstractions. It does not introduce a new public abstraction or global dependency.

Limitations

Runtime validation covers nspin=1, double precision, one MPI rank, one CUDA GPU, and the existing stm_bias=2 case. Multi-rank GPU LDOS, single-precision GPU input, other spin modes, other bias values, ROCm, Wannier90, and SOC paths were not runtime-tested.

Fixes part of #7770

@mohanchen mohanchen added the Feature Discussed The features will be discussed first but will not be implemented soon label Aug 8, 2026
@mohanchen
mohanchen requested review from chengleizheng and mohanchen and removed request for chengleizheng August 8, 2026 07:07
Comment thread source/source_io/module_dos/cal_ldos.h
Use a temporary CPU-double PW basis for post-SCF LDOS so synchronized host wavefunctions never enter CPU FFT routines through a GPU-configured basis. Keep the PW LDOS interface non-templated and enable the existing strict GPU regression case.
@CleverPhysician
CleverPhysician force-pushed the fix/pw-gpu-ldos-fft-dispatch branch from 844f9cb to 83f379f Compare August 11, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature Discussed The features will be discussed first but will not be implemented soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants