Post-mortems and analysis of real PyTorch training failures.
A real PyTorch bug (open since 2020, 25+ participants) produces NaN gradients when attn_mask and key_padding_mask together leave a row fully masked. We reproduce it, document NeuralDBG's blind spot on composite modules, and confirm the community workaround.
varlen_attn silently produces NaN gradients when extra padding tokens exceed cu_seqlens. Forward pass looks fine, backward explodes. NeuralDBG traces the NaN to qkv.weight. PR #188933 adds input validation that converts silent corruption into an explicit error — a real fix, not just a test.
torch.linalg.svdvals silently returns finite values for NaN input while svd correctly propagates NaN. This inconsistency causes silent gradient corruption in spectral normalization and condition number estimation. PR #188053 — reviewed by albanD (PyTorch collaborator).
F.normalize on a zero vector returns ~1,000,000,000,000 in backward pass instead of NaN. This happens naturally after dropout or masked attention. Loss looks normal while weights are silently corrupted. PR #188066 — 13 CI failures fixed, tests now pass.
CUDA nn.LSTM silently corrupts ALL samples when one sample has NaN input — a sample independence violation. NeuralDBG detects it with +24 anomaly gap, traces the causal chain, and the fix returns anomalies to 0. Perfect 0→24→0 detection.
MPS backend produces gradients 100x-100,000x too large — finite but wrong, invisible to NaN monitoring. NeuralDBG catches the NORMAL→EXPLODING gradient health transition. Full pipeline (detect→chain→fix→validate) achieves PASS.
Qwen3.5 experiences SDPA gradient explosion. Our only HuggingFace bug — demonstrates cross-ecosystem detection. PR #47024 was our first HF contribution.
torch.compile silently produces wrong gradients for atan2 — reported by @ezyang. Gap +24. Compiler bugs: no crash, no NaN, just wrong results.
torch.quantile with Inductor produces wrong gradients for tied values. Gap +16.
SDPA 32-bit offset overflow for long sequences. Silent corruption, critical as context windows scale.
A synthetic 6-layer MLP failure (vanishing gradients, then NaN) dissected layer by layer. Demonstrates how structured causal hypotheses would have saved hours of debugging. Illustrative scenario, not a real upstream bug.