[QST] Is this level of run-to-run variance expected when training HSTU retrieval on ML-20M?

[ngocbh]

What is your question?

When training the HSTU retrieval example on ML-20M (movielen_retrieval.gin) for 100 epochs, I see large run-to-run variance in eval metrics across identical config + identical seed runs. I'd like to understand whether this magnitude is expected, and whether there's a recommended path to more reproducible training.

Observed variance (same config, same seed=1234, 100 epochs, only re-launched)

setup	runs	final HR@10	peak HR@10	final loss
2×H100 (DP)	5	0.569 – 0.712 (spread ~0.14)	0.643 – 0.726	0.52 – 0.60
1×H100	6	0.649 – 0.759 (spread ~0.11)	~0.70 mean	0.55 – 0.75

Training is stable (smooth loss, no NaN/spikes) — this is run-to-run nondeterminism, not divergence. But the spread is large enough that comparing two single runs is unreliable.

Diagnostics I ran

• Data order is deterministic — tokens/iter match exactly across runs.
• Same-seed runs still diverge from ~iter 99 (per-step loss differs ~1%, compounding).
• torch.use_deterministic_algorithms(True) + CUBLAS_WORKSPACE_CONFIG=:4096:8 → no effect (runs still diverge), suggesting the nondeterminism is in custom fused kernels, not ATen.
• 1×GPU is tighter than 2×GPU but still nondeterministic → DP all-reduce is one contributor, not the only one.
• Switching the sparse embedding optimizer adam → row_wise_adagrad did not change the determinism.

My working hypothesis: the atomic-based backward of the fused kernels (FBGEMM TBE embedding backward for duplicate indices + HSTU CUTLASS attention backward) plus bf16 DP all-reduce make this inherently nondeterministic, and the small per-step differences are amplified by the small model + fixed LR (no warmup/decay).

Questions

1. Is this variance magnitude expected/normal for HSTU training, or does it indicate a problem in my setup?
2. Are there recommended settings for reproducible runs (e.g., a deterministic embedding/attention backward, or known-good flags)?
3. Does the validated Docker image (CUDA 13) behave more deterministically than a source build on CUDA 12.6?

Environment

• Install: from source (not Docker). The Dockerfile targets nvcr.io/nvidia/pytorch:26.05-py3 (CUDA 13); I built against the cluster's CUDA 12.6 toolkit instead.
• GPUs: NVIDIA A40 (sm_86) and H100 (sm_90), driver 570.211.01
• CUDA toolkit: 12.6 (nvcc V12.6.20) · gcc 12.2 · Python 3.10.13
• PyTorch: 2.12.1+cu126
• Packages: FBGEMM_GPU v1.5.0, TorchRec 1.5.0, Megatron-Core core_v0.13.1, fbgemm_gpu_hstu (jiayus-nvidia fork recsys-examples-v26.05)
• Config: movielen_retrieval.gin (4 layers, hidden 256, bf16, sampled-softmax 128 in-batch negatives, Adam lr 1e-3, no LR schedule), 100 epochs.

[shijieliu]

hi. @ngocbh thanks for your interest for recsys-examples!

About the deterministic issue, I believe your hypothesis is basically right. There are several sources that variance may come: embedding bwd reduction, the initialization of embedding may also introduce variance if you are using dynamicemb. And hstu bwd reduction and allreduce can also be non-deterministic. But from what i know, the torchrec FBGEMM should already supports deterministic with torch.use_deterministic_algorithms(True). However, i am not an expert of TorchRec so you may need to confirm with TorchRec team.

Are there recommended settings for reproducible runs (e.g., a deterministic embedding/attention backward, or known-good flags)?

For reproducible runs, you can start by setting DEMB_DETERMINISM_MODE =1 and enabling torch.use_deterministic_algorithms(True) to see if the non-determinism is resolved.
If the issue persists, you could try replacing DynamicEmbArgs with EmbeddingArgs to remove DynamicEmb from the pipeline and check whether that eliminates the variance.
If non-determinism still remains, you may need to isolate the source by checking each component individually and replacing any non-deterministic operations with deterministic alternatives.

Does the validated Docker image (CUDA 13) behave more deterministically than a source build on CUDA 12.6?

I dont think bumping cuda version can help here.