Nonlinear Initialization Methods for Low-Rank Neural Networks

Kiran Vodrahalli, Rakesh Shivanna, Maheswaran Sathiamoorthy +2

We propose a novel low-rank initialization framework for training low-rank deep neural networks -- networks where the weight parameters are re-parameterized by products of two low-…