Notes on Deep Learning Theory
arXiv:2012.05760
Abstract
These are the notes for the lectures that I was giving during Fall 2020 at the Moscow Institute of Physics and Technology (MIPT) and at the Yandex School of Data Analysis (YSDA). The notes cover some aspects of initialization, loss landscape, generalization, and a neural tangent kernel theory. While many other topics (e.g. expressivity, a mean-field theory, a double descent phenomenon) are missing in the current version, we plan to add them in future revisions.
arXiv comment68 pages
References in corpus (10)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Understanding deep learning requires rethinking generalization
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
- Uniform convergence may be unable to explain generalization in deep learning
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Depth Creates No Bad Local Minima
- Dynamics of Deep Neural Networks and Neural Tangent Hierarchy