Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't

Abstract

The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.

arXiv commentReview article. Feedback welcome

References in corpus (35)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• On Exact Computation with an Infinitely Wide Neural Net
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• On the Computational Efficiency of Training Neural Networks
• SGD Learns the Conjugate Kernel Class of the Network
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• The Gaussian equivalence of generative models for learning with shallow neural networks
• A Priori Estimates of the Population Risk for Residual Networks
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• Local minima in training of neural networks
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• The loss landscape of overparameterized neural networks
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• Representation formulas and pointwise properties for Barron functions
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Multiscale Adaptive Representation of Signals: I. The Basic Framework
• Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
• Loss Patterns of Neural Networks
• Poly-time universality and limitations of deep learning
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
• Global Convergence of Gradient Descent for Deep Linear Residual Networks
• A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern

Cited by in corpus (29)

• The Modern Mathematics of Deep Learning
• Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation
• Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
• Learning with invariances in random features and kernel models
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs
• A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equations
• Understanding Deep Learning via Decision Boundary
• Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration
• Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• Generalization and Memorization: The Bias Potential Model
• Recurrent Neural Networks for Partially Observed Dynamical Systems
• Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality
• Bayesian Inference Forgetting
• Conditioning of Random Feature Matrices: Double Descent and Generalization Error
• Reinforcement Learning with Function Approximation: From Linear to Nonlinear
• Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation
• Noisy Recurrent Neural Networks
• On the existence of optimal shallow feedforward networks with ReLU activation
• Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
• Robustness, Privacy, and Generalization of Adversarial Training
• A reduced order Schwarz method for nonlinear multiscale elliptic equations based on two-layer neural networks
• Embedding Principle of Loss Landscape of Deep Neural Networks
• A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions
• Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems
• Neural networks behave as hash encoders: An empirical study
• Stationary Density Estimation of Itô Diffusions Using Deep Learning
• Minimum complexity interpolation in random features models