Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

Abstract

Attempts from different disciplines to provide a fundamental understanding of deep learning have advanced rapidly in recent years, yet a unified framework remains relatively limited. In this article, we provide one possible way to align existing branches of deep learning theory through the lens of dynamical system and optimal control. By viewing deep neural networks as discrete-time nonlinear dynamical systems, we can analyze how information propagates through layers using mean field theory. When optimization algorithms are further recast as controllers, the ultimate goal of training processes can be formulated as an optimal control problem. In addition, we can reveal convergence and generalization properties by studying the stochastic dynamics of optimization algorithms. This viewpoint features a wide range of theoretical study from information bottleneck to statistical physics. It also provides a principled way for hyper-parameter tuning when optimal control theory is introduced. Our framework fits nicely with supervised learning and can be extended to other learning problems, such as Bayesian learning, adversarial training, and specific forms of meta learning, without efforts. The review aims to shed lights on the importance of dynamics and optimal control when developing deep learning theory.

Under Submission

References in corpus (29)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• A Convergence Theory for Deep Learning via Over-Parameterization
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Three Factors Influencing Minima in SGD
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Training Neural Networks with Local Error Signals
• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses
• Gradient descent aligns the layers of deep linear networks
• Deep Convolutional Networks as shallow Gaussian Processes
• The Emergence of Spectral Universality in Deep Networks
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Geometry of Optimization and Implicit Regularization in Deep Learning
• First-order Methods Almost Always Avoid Saddle Points
• Stochastic Thermodynamics of Learning
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• Where is the Information in a Deep Neural Network?
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Transport Analysis of Infinitely Deep Neural Network
• Defense Against the Dark Arts: An overview of adversarial example security research and future research directions
• Neural Networks with Cheap Differential Operators

Cited by in corpus (19)

• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Learn to Synchronize, Synchronize to Learn
• Physical deep learning based on optimal control of dynamical systems
• Continuous-in-Depth Neural Networks
• Towards a Systems Theory of Algorithms
• Towards Robust and Stable Deep Learning Algorithms for Forward Backward Stochastic Differential Equations
• Mean-field Langevin System, Optimal Control and Deep Neural Networks
• Finite-Time Convergence of Continuous-Time Optimization Algorithms via Differential Inclusions
• A Shooting Formulation of Deep Learning
• DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
• DDPNOpt: Differential Dynamic Programming Neural Optimizer
• SDE approximations of GANs training and its long-run behavior
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Second-Order Neural ODE Optimizer
• Go with the Flow: Adaptive Control for Neural ODEs
• A Dynamical Systems Approach for Convergence of the Bayesian EM Algorithm
• A priori guarantees of finite-time convergence for Deep Neural Networks
• Generative Adversarial Network: Some Analytical Perspectives
• Dynamic Game Theoretic Neural Optimizer