Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization

Aleksandra Irena Nowak, Łukasz Gniecki, Filip Szatkowski +1

Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which dete…