Neural gradients are near-lognormal: improved quantized and sparse training

Brian Chmiel, Liad Ben-Uri, Moran Shkolnik +3

While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning…