Bug or Feature${}^2$: Weight Drift, Activation Sparsity and Spikes

Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood…