The Modern Mathematics of Deep Learning

Abstract

We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.

arXiv commentA version of this review paper appears as a chapter in the book "Mathematical Aspects of Deep Learning" by Cambridge University Press

References in corpus (105)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Neural Machine Translation by Jointly Learning to Align and Translate
• Playing Atari with Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• Geometric deep learning: going beyond Euclidean data
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Solving high-dimensional partial differential equations using deep learning
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Training Very Deep Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Recent Trends in Deep Learning Based Natural Language Processing
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• On the Number of Linear Regions of Deep Neural Networks
• The Loss Surfaces of Multilayer Networks
• Understanding Deep Convolutional Networks
• Machine learning prediction errors better than DFT accuracy
• A Convergence Theory for Deep Learning via Over-Parameterization
• Benign Overfitting in Linear Regression
• fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
• Deep Neural Networks as Gaussian Processes
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• On Exact Computation with an Infinitely Wide Neural Net
• Train faster, generalize better: Stability of stochastic gradient descent
• Exploring Generalization in Deep Learning
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Inhomogeneous backflow transformations in quantum Monte Carlo calculations
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Solving Many-Electron Schrödinger Equation Using Deep Neural Networks
• The Power of Depth for Feedforward Neural Networks
• Provable approximation properties for deep neural networks
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• Spectrally-normalized margin bounds for neural networks
• The loss surface of deep and wide neural networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Scaling description of generalization with number of parameters in deep learning
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Fantastic Generalization Measures and Where to Find Them
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Interactions between Large Molecules: Puzzle for Reference Quantum-Mechanical Methods
• Why Deep Neural Networks for Function Approximation?
• On multilevel Picard numerical approximations for high-dimensional nonlinear parabolic partial differential equations and high-dimensional nonlinear backward stochastic differential equations
• Optimal approximation of continuous functions by very deep ReLU networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
• Representation Benefits of Deep Feedforward Networks
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Uniform convergence may be unable to explain generalization in deep learning
• Are All Layers Created Equal?
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Solving the Kolmogorov PDE by means of deep learning
• Loss of regularity for Kolmogorov equations
• Adversarial Regularizers in Inverse Problems
• Norm-Based Capacity Control in Neural Networks
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• The phase diagram of approximation rates for deep neural networks
• Strong error analysis for stochastic gradient descent optimization algorithms
• Deep ReLU network approximation of functions on a manifold
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Analysis of time-frequency scattering transforms
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
• On the Implicit Bias of Dropout
• Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks
• Topology and Geometry of Half-Rectified Network Optimization
• Multiple Descent: Design Your Own Generalization Curve
• Convergence of Gradient Descent on Separable Data
• Representation formulas and pointwise properties for Barron functions
• Learning without Concentration
• Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities
• Expressivity of Deep Neural Networks
• What causes the test error? Going beyond bias-variance via ANOVA
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Deep Limits of Residual Neural Networks
• Directional convergence and alignment in deep learning
• The gap between theory and practice in function approximation with deep neural networks
• Deep Learning via Dynamical Systems: An Approximation Perspective
• Entropy and the Combinatorial Dimension
• Sparse Neural Networks Topologies
• Neural network approximation and estimation of classifiers with classification boundary in a Barron class
• Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spaces
• A priori estimates for classification problems using neural networks
• The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training
• Solving the electronic Schrödinger equation for multiple nuclear geometries with weight-sharing deep neural networks
• Quantifying Model Uncertainty in Inverse Problems via Bayesian Deep Gradient Descent
• Elementary superexpressive activations
• Deep neural network approximation for high-dimensional parabolic Hamilton-Jacobi-Bellman equations

Cited by in corpus (17)

• Incorporating Domain Knowledge into Deep Neural Networks
• A Review of Some Techniques for Inclusion of Domain-Knowledge into Deep Neural Networks
• Combining physics-based and data-driven models: advancing the frontiers of research with Scientific Machine Learning
• Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performance
• Limitations of Deep Learning for Inverse Problems on Digital Hardware
• Learning-informed parameter identification in nonlinear time-dependent PDEs
• Emergence of Concepts in DNNs?
• Towards a population-informed approach to the definition of data-driven models for structural dynamics
• Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality
• On generalization bounds for deep networks based on loss surface implicit regularization
• Rosenblatt's first theorem and frugality of deep learning
• A machine learning approach to predict L-edge x-ray absorption spectra of light transition metal ion compounds
• Neural variance reduction for stochastic differential equations
• A Low Rank Neural Representation of Entropy Solutions
• ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computation
• Neural Networks in Fréchet spaces
• First passage time in space-dependent stochastic resetting