The Dynamics of Learning: A Random Matrix Approach

Abstract

Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neural nets work so well today. In this paper, we introduce a random matrix-based framework to analyze the learning dynamics of a single-layer linear network on a binary classification problem, for data of simultaneously large dimension and size, trained by gradient descent. Our results provide rich insights into common questions in neural nets, such as overfitting, early stopping and the initialization of training, thereby opening the door for future studies of more elaborate structures and models appearing in today's neural networks.

arXiv comment14 pages (with Supplementary Material), 7 figures, ICML 2018

Cited by in corpus (13)

• Scaling description of generalization with number of parameters in deep learning
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
• When Does Preconditioning Help or Hurt Generalization?
• High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Loss
• Products of Complex Rectangular and Hermitian Random Matrices
• Hard Edge Statistics of Products of Pólya Ensembles and Shifted GUE's
• Two-way kernel matrix puncturing: towards resource-efficient PCA and spectral clustering
• A Random Matrix Perspective on Random Tensors
• Thompson Sampling for Noncompliant Bandits
• Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
• Random matrices in service of ML footprint: ternary random features with no performance loss
• Understanding the wiring evolution in differentiable neural architecture search