A Differential Game Theoretic Neural Optimizer for Training Residual Networks
arXiv:2007.08880
Abstract
Connections between Deep Neural Networks (DNNs) training and optimal control theory has attracted considerable attention as a principled tool of algorithmic design. Differential Dynamic Programming (DDP) neural optimizer is a recently proposed method along this line. Despite its empirical success, the applicability has been limited to feedforward networks and whether such a trajectory-optimization inspired framework can be extended to modern architectures remains unclear. In this work, we derive a generalized DDP optimizer that accepts both residual connections and convolution layers. The resulting optimal control representation admits a game theoretic perspective, in which training residual networks can be interpreted as cooperative trajectory optimization on state-augmented dynamical systems. This Game Theoretic DDP (GT-DDP) optimizer enjoys the same theoretic connection in previous work, yet generates a much complex update rule that better leverages available information during network propagation. Evaluation on image classification datasets (e.g. MNIST and CIFAR100) shows an improvement in training convergence and variance reduction over existing methods. Our approach highlights the benefit gained from architecture-aware optimization.
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- A guide to convolution arithmetic for deep learning
- Opening the Black Box of Deep Neural Networks via Information
- Deep Reinforcement Learning that Matters
- On Convergence and Stability of GANs
- The Numerics of GANs
- Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
- The Mechanics of n-Player Differentiable Games
- Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control
- Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
- You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Neuron Shapley: Discovering the Responsible Neurons
- An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
- Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
- Analysing Neural Network Topologies: a Game Theoretic Approach
- Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examples
- A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
- Splitting Steepest Descent for Growing Neural Architectures
- Stochastic Training of Residual Networks: a Differential Equation Viewpoint