Geometric compression of invariant manifolds in neural nets

Abstract

We study how neural networks compress uninformative input space in models where data lie in $d$ dimensions, but whose label only vary within a linear manifold of dimension $d_{\parallel }<d$. We show that for a one-hidden layer network initialized with infinitesimal weights (i.e. in the feature learning regime) trained with gradient descent, the first layer of weights evolve to become nearly insensitive to the $d_{\perp }=d-d_{\parallel }$ uninformative directions. These are effectively compressed by a factor $\lambda \sim \sqrt{p}$, where $p$ is the size of the training set. We quantify the benefit of such a compression on the test error $\epsilon$. For large initialization of the weights (the lazy training regime), no compression occurs and for regular boundaries separating labels we find that $\epsilon \sim p^{-\beta }$, with ${\beta }_{\text{Lazy}}=d/(3d-2)$. Compression improves the learning curves so that ${\beta }_{\text{Feature}}=(2d-1)/(3d-2)$ if $d_{\parallel }=1$ and ${\beta }_{\text{Feature}}=(d+d_{\perp }/2)/(3d-2)$ if $d_{\parallel }>1$. We test these predictions for a stripe model where boundaries are parallel interfaces ($d_{\parallel }=1$) as well as for a cylindrical boundary ($d_{\parallel }=2$). Next we show that compression shapes the Neural Tangent Kernel (NTK) evolution in time, so that its top eigenvectors become more informative and display a larger projection on the labels. Consequently, kernel learning with the frozen NTK at the end of training outperforms the initial NTK. We confirm these predictions both for a one-hidden layer FC network trained on the stripe model and for a 16-layers CNN trained on MNIST, for which we also find ${\beta }_{\text{Feature}}>{\beta }_{\text{Lazy}}$.

References in corpus (9)

• Understanding Deep Convolutional Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Estimating the intrinsic dimension of datasets by a minimal neighborhood information
• Scaling description of generalization with number of parameters in deep learning
• Statistical Mechanics of Support Vector Networks
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Disentangling feature and lazy training in deep neural networks
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

Cited by in corpus (3)

• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Finite-time Lyapunov exponents of deep neural networks
• Relative stability toward diffeomorphisms indicates performance in deep nets