Harnessing Orthogonality to Train Low-Rank Neural Networks

Daniel Coquelin, Katharina Flügel, Marie Weiel +4

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals tha…