Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks
arXiv:2003.12895
Abstract
We prove that a single step of gradient decent over depth two network, with hidden neurons, starting from orthogonal initialization, can memorize independent and randomly labeled Gaussians in . The result is valid for a large class of activation functions, which includes the absolute value.
References in corpus (18)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
- Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- SGD Learns the Conjugate Kernel Class of the Network
- Diverse Neural Network Learns True Target Functions
- Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
- Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
- An Improved Analysis of Training Over-parameterized Deep Neural Networks
- SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
- Neural Networks Learning and Memorization with (almost) no Over-Parameterization
- Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently