Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Abstract

Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures. In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been studied in mathematics and physics with rich theoretical and empirical success. From this interpretation, we develop a theoretical framework on stability and reversibility of deep neural networks, and derive three reversible neural network architectures that can go arbitrarily deep in theory. The reversibility property allows a memory-efficient implementation, which does not need to store the activations for most hidden layers. Together with the stability of our architectures, this enables training deeper networks using only modest computational resources. We provide both theoretical analyses and empirical results. Experimental results demonstrate the efficacy of our architectures against several strong baselines on CIFAR-10, CIFAR-100 and STL-10 with superior or on-par state-of-the-art performance. Furthermore, we show our architectures yield superior results when trained using fewer training data.

arXiv commentAccepted at AAAI 2018

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• The Microsoft 2016 Conversational Speech Recognition System
• Deep Networks with Stochastic Depth
• PDE-Net: Learning PDEs from Data
• FiLM: Visual Reasoning with a General Conditioning Layer
• Convolutional Clustering for Unsupervised Learning
• Multi-level Residual Networks from Dynamical Systems View
• Why are deep nets reversible: A simple theory, with implications for training
• Learning across scales - A multiscale method for Convolution Neural Networks

Cited by in corpus (26)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Neural Ordinary Differential Equations
• Inverting Gradients -- How easy is it to break privacy in federated learning?
• A deep learning enabler for non-intrusive reduced order modeling of fluid flows
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Multi-level Residual Networks from Dynamical Systems View
• GFINNs: GENERIC Formalism Informed Neural Networks for Deterministic and Stochastic Dynamical Systems
• Don't Unroll Adjoint: Differentiating SSA-Form Programs
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• Competitive Inner-Imaging Squeeze and Excitation for Residual Network
• Monge-Ampère Flow for Generative Modeling
• Robust learning with implicit residual networks
• State Space Representations of Deep Neural Networks
• Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images
• Optimizing Neural Networks via Koopman Operator Theory
• REX: Revisiting Budgeted Training with an Improved Schedule
• Generative Adversarial Forests for Better Conditioned Adversarial Learning
• Functional Gradient Boosting based on Residual Network Perception
• Unsupervised Reservoir Computing for Solving Ordinary Differential Equations
• Identity Connections in Residual Nets Improve Noise Stability
• Memory-Efficient Differentiable Transformer Architecture Search
• Differential equations as models of deep neural networks
• Machine Learning and Control Theory
• Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families
• Vector Field Neural Networks
• Implicit Euler ODE Networks for Single-Image Dehazing