Gradient Explosion: When Your Model Produces 100,000x Gradients

Anomalies detected: 0 (healthy) → 24 (bug) → 1 (fixed)

1. The Bug

The MPS backend (Apple Silicon GPU) produces gradients that are 100x–100,000x too large. The gradients are finite (no NaN), so standard monitoring doesn't catch them — the model silently converges to wrong weights.

Upstream: pytorch#177116 — OPEN, labeled module: mps, triaged.

2. Reproduction (CPU simulation)

model = nn.Sequential(nn.Linear(8,32), nn.ReLU(), nn.Linear(32,16), nn.ReLU(), nn.Linear(16,2))
x_corrupted = torch.randn(4, 8) * 100  # simulates MPS corruption
out = model(x_corrupted); out.sum().backward()
# Gradients: ~100-100,000x too large — but still finite!

3. NeuralDBG Diagnosis

With the DeepMLP architecture (12 layers, skip connections), NeuralDBG captures 24 anomalies vs 0 in the healthy baseline:

• data_anomaly at LayerNorm_0: distribution_shift
• gradient_health_transition at Linear_head: NORMAL → EXPLODING
• optimizer_instability at optimizer: diverging

4. Why Standard Tools Miss This

5. End-to-End Pipeline

The full NeuralSuite closed loop works on this bug:

[1/4] Healthy: 0 anomalies
[2/4] Bug: 24 anomalies (DETECTED, gap +24)
      Chain: data_anomaly → gradient[exploding] → optimizer[diverging]
[3/4] Fix: normal data restored
[4/4] Validate: 1 anomaly (RESOLVED)
VERDICT: PASS

Causal Chain

Detection Metrics