Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Abstract

We show that the gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given $\tilde{O}(d{r}^2)$ random linear measurements of a rank $r$ positive semidefinite matrix $X^{\star }$, we can recover $X^{\star }$ by parameterizing it by $U{U}^{\top }$ with $U\in {\mathbb{R}}^{d\times d}$ and minimizing the squared loss, even if $r\ll d$. We prove that starting from a small initialization, gradient descent recovers $X^{\star }$ in $\tilde{O}(\sqrt{r})$ iterations approximately. The results solve the conjecture of Gunasekar et al.'17 under the restricted isometry property. The technique can be applied to analyzing neural networks with one-hidden-layer quadratic activations with some technical modifications.

COLT 2018 best paper; fixed minor missing steps in the previous version

References in corpus (6)

• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
• On the Computational Efficiency of Training Neural Networks
• On the Universality of Online Mirror Descent

Cited by in corpus (4)

• Reconciling modern machine learning practice and the bias-variance trade-off
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• When Does Preconditioning Help or Hurt Generalization?
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex