Recent advances in deep learning theory

Abstract

Deep learning is usually described as an experiment-driven field under continuous criticizes of lacking theoretical foundations. This problem has been partially fixed by a large volume of literature which has so far not been well organized. This paper reviews and organizes the recent advances in deep learning theory. The literature is categorized in six groups: (1) complexity and capacity-based approaches for analyzing the generalizability of deep learning; (2) stochastic differential equations and their dynamic systems for modelling stochastic gradient descent and its variants, which characterize the optimization and generalization of deep learning, partially inspired by Bayesian inference; (3) the geometrical structures of the loss landscape that drives the trajectories of the dynamic systems; (4) the roles of over-parameterization of deep neural networks from both positive and negative perspectives; (5) theoretical foundations of several special structures in network architectures; and (6) the increasingly intensive concerns in ethics and security and their relationships with generalizability.

References in corpus (58)

• A Survey on Deep Learning in Medical Image Analysis
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Deep Learning with Differential Privacy
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Equality of Opportunity in Supervised Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Normalizing Flows: An Introduction and Review of Current Methods
• Renyi Differential Privacy
• An Analysis of Deep Neural Network Models for Practical Applications
• Theoretically Principled Trade-off between Robustness and Accuracy
• The Loss Surfaces of Multilayer Networks
• Group Equivariant Convolutional Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Machine Learning Energies of 2 M Elpasolite (ABC${}_2$D${}_6$) Crystals
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Three Factors Influencing Minima in SGD
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Differential Privacy as a Mutual Information Constraint
• Learning Non-Discriminatory Predictors
• Adversarial Attack on Graph Structured Data
• Motivating the Rules of the Game for Adversarial Example Research
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Optimization for deep learning: theory and algorithms
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Avoiding pathologies in very deep networks
• Depth Creates No Bad Local Minima
• Dynamical Mass Measurements of Contaminated Galaxy Clusters Using Machine Learning
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Universal approximations of permutation invariant/equivariant functions by deep neural networks
• Generalization Bounds via Information Density and Conditional Information Density
• Nested Variational Compression in Deep Gaussian Processes
• Stability and Convergence Trade-off of Iterative Optimization Algorithms
• Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
• Lower Bounds on Adversarial Robustness from Optimal Transport
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Theoretical Analysis of Adversarial Learning: A Minimax Approach
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Small nonlinearities in activation functions create bad local minima in neural networks
• Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
• Adversarial Risk Bounds via Function Transformation
• On the Generalization Properties of Differential Privacy
• An Information-Theoretic View for Deep Learning
• On the role of data in PAC-Bayes bounds
• On Generalization Bounds of a Family of Recurrent Neural Networks
• More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
• Piecewise linear activations substantially shape the loss surfaces of neural networks
• Estimating Cosmological Parameters from the Dark Matter Distribution
• Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
• Generalization Bounds for Convolutional Neural Networks
• Toward Better Generalization Bounds with Locally Elastic Stability
• Capacity Bounded Differential Privacy
• Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization

Cited by in corpus (9)

• A Survey on Kolmogorov-Arnold Network
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• Understanding Deep Learning via Decision Boundary
• Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?
• Strength of Minibatch Noise in SGD
• Bayesian Inference Forgetting
• Robustness, Privacy, and Generalization of Adversarial Training
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Neural networks behave as hash encoders: An empirical study