Parameter settings in parallel band structure calculation

[Ljiamin195]

Dear the developer of HamGNN,

It took me 5 to 6 days to perform a parallel band structure calculation for a system containing over 3,000 atoms. I only set 40 K-points. Is this computational efficiency considered normal? If not, what could be the potential reasons for such a long runtime?

The attached images show the input settings, submission script, and output, respectively.

Best regards,
Jiamin Luo

[QuantumLab-ZY]

Dear Jiamin,

Thank you for reaching out and for using HamGNN!

What's going wrong:

band_cal_parallel is a pure MPI program, meaning it relies on multiple MPI processes (not multi-threading) for parallelism. In your current submission script, you have:

--cpus-per-task=9
export OMP_NUM_THREADS=1

This means each MPI process is allocated 9 CPU cores but only uses 1 of them, leaving ~89% of your allocated resources idle. This also significantly limits the number of MPI processes that can be launched on each node, which directly reduces parallel efficiency.

Recommended fix:

Please modify your SLURM script as follows:

#SBATCH --cpus-per-task=1          # Each MPI process needs only 1 core
#SBATCH --ntasks-per-node=<N>      # Set <N> to the total number of cores per node (e.g., 36, 48, 64, depending on your cluster)

A simple way to think about it:

• Pure MPI programs → maximize the number of tasks (processes), minimize CPUs per task (set to 1).

With this change, you should see a dramatic speedup, as you will be utilizing all available CPU cores effectively.

Additional tips for large systems (3,000+ atoms):

You can also try increasing the number of nodes (--nodes) to further distribute the workload across more MPI processes.

Best,
Yang Zhong

[Ljiamin195]

Dear the developer of HamGNN,

Thank you very much for your reply!

Following your advice, I’ve re-calculated using 64 cores on a single node (Actually, the system consists of over 4,000 atoms—I misstated the number previously.), and it still took three days. Is this speed normal?

For my next step, if I want to calculate a system with over ten thousand atoms, is increasing the number of nodes the only way to improve efficiency?

Best,
Jiamin Luo

[QuantumLab-ZY]

Regarding the runtime for your 4,000+ atom system:

Yes, the speed you're experiencing is within the expected range. For a system of over 4,000 atoms, the diagonalization of the Hamiltonian matrix is the primary computational bottleneck — its cost scales roughly as O(N³) with respect to the matrix dimension. So a three-day runtime on 64 cores for a system of this size is not unusual.

Regarding scaling to systems with over 10,000 atoms:

Increasing the number of cores is indeed one of the most straightforward ways to improve efficiency. For a system of that size, I would recommend using 1,000 or more cores across multiple nodes for the diagonalization step.

[Ljiamin195]

While using band_cal_parallel to calculate a larger system, I encountered the following ScaLAPACK error:
PCHEEVD parameter number 14 had an illegal value

This is a 32-bit integer overflow issue. For my system, which contains approximately 77,000 orbitals or more, the required LWORK size significantly exceeds the 32-bit integer limit (2.14 billion).

Since the currently provided mpitool package is compiled with LP64, I need to recompile it and link it against Intel MKL ILP64 (libmkl_intel_ilp64.so) to support 64-bit integers.

Could you please provide the source code for mpitool ？Alternatively, would it be possible for you to directly release a wheel package compiled with ILP64?

Thank you!

[QuantumLab-ZY]

mpitool is compiled with ILP64, so large-scale calculations should not be a problem — we have successfully diagonalized a GaAs defect structure with 13,000 atoms across 2,000 CPU cores without any issues.

Based on our experience, the error you're encountering may not reflect the actual root cause. It is likely that running on a single node leads to insufficient memory, which in turn produces a misleading error message.

[Ljiamin195]

Dear the developer of HamGNN,

Thank you for your response！

I increased the number of nodes and cores and successfully calculated a system with over 7,000 atoms. However, when calculating a system with more than 10,000 atoms, I encountered the following error, even though I used 16 nodes, each with 64 cores and 512 GB of memory. I'm not sure what's causing this. Could you give me some advice?

File "/public/home/luojiamin/miniconda3/envs/band_cal/lib/python3.9/site-packages/HamTool/band_cal_parallel.py", line 818, in band_cal_nonsoc mpitool.general_matrix_solve(1, vbm_band-num_wfns, vbm_band+num_wfns, glob_wfn, dist_k, HK_d, SK_d, eigenvec0, eigenval)
RuntimeError: Error in Cholesky.

Best regards,
Jiamin Luo

[QuantumLab-ZY]

Hi Luo,

This error likely indicates that the S matrix is not positive definite. As a first troubleshooting step, I'd recommend trying the following:

1. Vary the number of parallel nodes and resubmit the job several times.
2. Check whether the same error persists across different configurations.

This will help determine whether the issue is caused by numerical instability related to parallelization or if it points to a more fundamental problem with the input data/parameters.

Good luck,
Yang Zhong

[Ljiamin195]

Dear Prof. Zhong,

I've tried different numbers of nodes (8, 10, 12, and 16 nodes), but they all threw the same error. I also checked my structure, and the minimum distance between atoms is about 1.6 Å, so there shouldn't be the problem. Additionally, the parameter settings are completely consistent with my previous successful calculations.

Given this, could there be any other reasons causing the S matrix to lose its positive definite property in such a large system?

Best regards,
Jiamin Luo

[QuantumLab-ZY]

Although this is rare, it is indeed possible for some large structures to produce a non-positive definite overlap (S) matrix in OpenMX. I have encountered this issue before and verified it by diagonaling the overlap matrix using the serial diagonalizer as well.

Here are two possible workarounds:

1. Increase the energy cutoff in your openmx.dat input file. A higher cutoff energy improves the accuracy of the overlap matrix calculation, which may resolve the non-positive definiteness.
2. Rebuild a structure of similar scale and re-run the calculation.

Hope this helps!