Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training

Abstract

Deep learning algorithms are responsible for a technological revolution in a variety of tasks including image recognition or Go playing. Yet, why they work is not understood. Ultimately, they manage to classify data lying in high dimension -- a feat generically impossible due to the geometry of high dimensional space and the associated curse of dimensionality. Understanding what kind of structure, symmetry or invariance makes data such as images learnable is a fundamental challenge. Other puzzles include that (i) learning corresponds to minimizing a loss in high dimension, which is in general not convex and could well get stuck bad minima. (ii) Deep learning predicting power increases with the number of fitting parameters, even in a regime where data are perfectly fitted. In this manuscript, we review recent results elucidating (i,ii) and the perspective they offer on the (still unexplained) curse of dimensionality paradox. We base our theoretical discussion on the $(h,\alpha )$ plane where $h$ is the network width and $\alpha$ the scale of the output of the network at initialization, and provide new systematic measures of performance in that plane for MNIST and CIFAR 10. We argue that different learning regimes can be organized into a phase diagram. A line of critical points sharply delimits an under-parametrised phase from an over-parametrized one. In over-parametrized nets, learning can operate in two regimes separated by a smooth cross-over. At large initialization, it corresponds to a kernel method, whereas for small initializations features can be learnt, together with invariants in the data. We review the properties of these different phases, of the transition separating them and some open questions. Our treatment emphasizes analogies with physical systems, scaling arguments and the development of numerical observables to quantitatively test these results empirically.

References in corpus (51)

• Playing Atari with Deep Reinforcement Learning
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Theoretical perspective on the glass transition and amorphous materials
• Jamming at Zero Temperature and Zero Applied Stress: the Epitome of Disorder
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• The Loss Surfaces of Multilayer Networks
• Object Detectors Emerge in Deep Scene CNNs
• Understanding Deep Convolutional Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Fractal free energy landscapes in structural glasses
• Deep Learning Scaling is Predictable, Empirically
• An Empirical Evaluation of Deep Learning on Highway Driving
• Effects of compression on the vibrational modes of marginally jammed solids
• On the Rigidity of Amorphous Solids
• On Lazy Training in Differentiable Programming
• Universal Spectrum of Normal Modes in Low-Temperature Glasses: an Exact Solution
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Stress Propagation through Frictionless Granular Material
• A unified framework for non-Brownian suspension flows and soft amorphous solids
• Scaling description of generalization with number of parameters in deep learning
• A Landscape Analysis of Constraint Satisfaction Problems
• Stability at Random Close Packing
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Jamming in Systems Composed of Frictionless Ellipse-Shaped Particles
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• Scaling of phononic transport with connectivity in amorphous solids
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Unified Theory of Inertial Granular Flows and Non-Brownian Suspensions
• Perspective: Energy Landscapes for Machine Learning
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Universality of jamming of non-spherical particles
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Theory of the Jamming Transition at Finite Temperature
• Jamming in multilayer supervised learning models
• Asymptotics of Wide Networks from Feynman Diagrams
• Dimensionality compression and expansion in Deep Neural Networks
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
• The loss landscape of overparameterized neural networks
• Critical jammed phase of the linear perceptron
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Neural Kernels Without Tangents
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
• Minnorm training: an algorithm for training over-parameterized deep neural networks
• Stuck in a What? Adventures in Weight Space
• The asymptotic spectrum of the Hessian of DNN throughout training

Cited by in corpus (1)

• Relative stability toward diffeomorphisms indicates performance in deep nets