Gradient-based Weight Density Balancing for Robust Dynamic Sparse Training

Mathias Parger, Alexander Ertl, Paul Eibensteiner +3

Training a sparse neural network from scratch requires optimizing connections at the same time as the weights themselves. Typically, the weights are redistributed after a predefine…