On the Impact of Stable Ranks in Deep Nets

Abstract

A recent line of work has established intriguing connections between the generalization/compression properties of a deep neural network (DNN) model and the so-called layer weights' stable ranks. Intuitively, the latter are indicators of the effective number of parameters in the net. In this work, we address some natural questions regarding the space of DNNs conditioned on the layers' stable rank, where we study feed-forward dynamics, initialization, training and expressivity. To this end, we first propose a random DNN model with a new sampling scheme based on stable rank. Then, we show how feed-forward maps are affected by the constraint and how training evolves in the overparametrized regime (via Neural Tangent Kernels). Our results imply that stable ranks appear layerwise essentially as linear factors whose effect accumulates exponentially depthwise. Moreover, we provide empirical analysis suggesting that stable rank initialization alone can lead to convergence speed ups.

arXiv comment24 pages, 8 figures, comments welcome!

References in corpus (6)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Integration with respect to the Haar measure on unitary, orthogonal and symplectic group
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Gradient Descent Happens in a Tiny Subspace
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
• Emergent properties of the local geometry of neural loss landscapes