Mathematics of Deep Learning

Abstract

Recently there has been a dramatic increase in the performance of recognition systems due to the introduction of deep architectures for representation learning and classification. However, the mathematical reasons for this success remain elusive. This tutorial will review recent work that aims to provide a mathematical justification for several properties of deep networks, such as global optimality, geometric stability, and invariance of the learned representations.

References in corpus (15)

• Geometric deep learning: going beyond Euclidean data
• The information bottleneck method
• Wide Residual Networks
• Opening the Black Box of Deep Neural Networks via Information
• The Loss Surfaces of Multilayer Networks
• Understanding Deep Convolutional Networks
• Deep Variational Information Bottleneck
• Robust Large Margin Deep Neural Networks
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• Information Dropout: Learning Optimal Representations Through Noisy Computation
• On Invariance and Selectivity in Representation Learning

Cited by in corpus (27)

• Model based learning for accelerated, limited-view 3D photoacoustic tomography
• Optimization for deep learning: theory and algorithms
• A Survey on Neural Network Interpretability
• From Symmetry to Geometry: Tractable Nonconvex Problems
• Robust Conditional Generative Adversarial Networks
• Opening the black box of deep learning
• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
• Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation
• Provably scale-covariant continuous hierarchical networks based on scale-normalized differential expressions coupled in cascade
• Information Bottleneck and its Applications in Deep Learning
• Combining Image Features and Patient Metadata to Enhance Transfer Learning
• Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Cooling down stochastic differential equations: almost sure convergence
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Translating Diffusion, Wavelets, and Regularisation into Residual Networks
• A Reflection on Learning from Data: Epistemology Issues and Limitations
• High-dimensional Neural Feature Design for Layer-wise Reduction of Training Cost
• Stochastic Conditional Generative Networks with Basis Decomposition
• R3Net: Random Weights, Rectifier Linear Units and Robustness for Artificial Neural Network
• Error bounds for PDE-regularized learning
• Scientific Calculator for Designing Trojan Detectors in Neural Networks
• Enhanced Transfer Learning Through Medical Imaging and Patient Demographic Data Fusion
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions