A Selective Overview of Deep Learning

Abstract

Deep learning has arguably achieved tremendous success in recent years. In simple words, deep learning uses the composition of many nonlinear functions to model the complex dependency between input features and labels. While neural networks have a long history, recent advances have greatly improved their performance in computer vision, natural language processing, etc. From the statistical and scientific perspective, it is natural to ask: What is deep learning? What are the new characteristics of deep learning, compared with classical methods? What are the theoretical foundations of deep learning? To answer these questions, we introduce common neural network models (e.g., convolutional neural nets, recurrent neural nets, generative adversarial nets) and training techniques (e.g., stochastic gradient descent, dropout, batch normalization) from a statistical point of view. Along the way, we highlight new characteristics of deep learning (including depth and over-parametrization) and explain their practical and theoretical benefits. We also sample recent results on theories of deep learning, many of which are only suggestive. While a complete understanding of deep learning remains elusive, we hope that our perspectives and discussions serve as a stimulus for new statistical research.

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• How transferable are features in deep neural networks?
• On the Convergence of Adam and Beyond
• Understanding Neural Networks Through Deep Visualization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Generative Moment Matching Networks
• Fisher Lecture: Dimension Reduction in Regression
• Dropout Training as Adaptive Regularization
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• The power of deeper networks for expressing natural functions
• Learning Functions: When Is Deep Better Than Shallow
• Hierarchical interpretations for neural network predictions
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Risk and parameter convergence of logistic regression
• A Priori Estimates of the Population Risk for Residual Networks
• High probability generalization bounds for uniformly stable algorithms with nearly optimal rate
• Approximability of Discriminators Implies Diversity in GANs
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
• Robust Estimation and Generative Adversarial Nets
• Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization

Cited by in corpus (19)

• Social physics
• On Exact Computation with an Infinitely Wide Neural Net
• Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
• Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
• A Theoretical Analysis of Deep Q-Learning
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• Learning the Relationship between Galaxies Spectra and their Star Formation Histories using Convolutional Neural Networks and Cosmological Simulations
• The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent
• Forbidden knowledge in machine learning -- Reflections on the limits of research and publication
• On Representing (Anti)Symmetric Functions
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• Fiedler Regularization: Learning Neural Networks with Graph Sparsity
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Variational Transport: A Convergent Particle-BasedAlgorithm for Distributional Optimization
• Provably Training Overparameterized Neural Network Classifiers with Non-convex Constraints
• Detecting conflicting summary statistics in likelihood-free inference
• A Probabilistic Approach to Neural Network Pruning
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning