Implicit Regularization via Neural Feature Alignment
arXiv:2008.00938
Abstract
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.
arXiv commentAISTATS 2021
References in corpus (25)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Shortcut Learning in Deep Neural Networks
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Algorithms for Learning Kernels Based on Centered Alignment
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- On the Spectral Bias of Neural Networks
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
- Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
- An Investigation of Why Overparameterization Exacerbates Spurious Correlations
- Geometry of Optimization and Implicit Regularization in Deep Learning
- A Modern Take on the Bias-Variance Tradeoff in Neural Networks
- Towards Understanding the Spectral Bias of Deep Learning
- Kernel and Rich Regimes in Overparametrized Models
- Stiffness: A New Perspective on Generalization in Neural Networks
- An analytic theory of generalization dynamics and transfer learning in deep linear networks
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
- Pathological spectra of the Fisher information metric and its variants in deep neural networks