The Reversible Residual Network: Backpropagation Without Storing Activations

Abstract

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Density estimation using Real NVP
• Revisiting Distributed Synchronous SGD
• Training Deep Nets with Sublinear Memory Cost
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Xception: Deep Learning with Depthwise Separable Convolutions
• Neural Machine Translation in Linear Time
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
• Decoupled Neural Interfaces using Synthetic Gradients

Cited by in corpus (92)

• Normalizing Flows: An Introduction and Review of Current Methods
• Momentum Contrast for Unsupervised Visual Representation Learning
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Neural Ordinary Differential Equations
• Hamiltonian Neural Networks
• Large Scale Adversarial Representation Learning
• Normalizing Flows for Probabilistic Modeling and Inference
• A Review on Deep Learning in Medical Image Reconstruction
• Glow: Generative Flow with Invertible 1x1 Convolutions
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Invertible Residual Networks
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• WINNet: Wavelet-inspired Invertible Network for Image Denoising
• A Practical Survey on Faster and Lighter Transformers
• Yao.jl: Extensible, Efficient Framework for Quantum Algorithm Design
• MgNet: A Unified Framework of Multigrid and Convolutional Neural Network
• Streaming convolutional neural networks for end-to-end learning with multi-megapixel images
• Inference Suboptimality in Variational Autoencoders
• Bayesian Layers: A Module for Neural Network Uncertainty
• Multi-level Residual Networks from Dynamical Systems View
• Excessive Invariance Causes Adversarial Vulnerability
• i-RevNet: Deep Invertible Networks
• Efficient Modelling of Trivializing Maps for Lattice ${\varphi }^4$ Theory Using Normalizing Flows: A First Look at Scalability
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• FakeNews: GAN-based generation of realistic 3D volumetric data -- A systematic review and taxonomy
• Improving GANs Using Optimal Transport
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
• Low-Memory Neural Network Training: A Technical Report
• InversionNet3D: Efficient and Scalable Learning for 3D Full Waveform Inversion
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• ETC: Encoding Long and Structured Inputs in Transformers
• Neural Spline Flows
• Koopman operator learning using invertible neural networks
• Transport Analysis of Infinitely Deep Neural Network
• LoCo: Local Contrastive Representation Learning
• Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
• Backprop with Approximate Activations for Memory-efficient Network Training
• Q${}^2$Chemistry: A quantum computation platform for quantum chemistry
• Robust learning with implicit residual networks
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• Learning for routing: A guided review of recent developments and future directions
• Tensor2Tensor for Neural Machine Translation
• Challenges in Domain-Specific Abstractive Summarization and How to Overcome them
• Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory
• Neural Invertible Variable-degree Optical Aberrations Correction
• Training Invertible Neural Networks as Autoencoders
• p-Meta: Towards On-device Deep Model Adaptation
• Globally Injective ReLU Networks
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Layer-Parallel Training of Deep Residual Neural Networks
• Memory Optimization for Deep Networks
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• End-to-end Memory-Efficient Reconstruction for Cone Beam CT
• Training Generative Reversible Networks
• Training convolutional neural networks with megapixel images
• Randomized Automatic Differentiation
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks
• Aggregated Momentum: Stability Through Passive Damping
• Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images
• Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
• Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory
• Sparsely Aggregated Convolutional Networks
• UFO-BLO: Unbiased First-Order Bilevel Optimization
• Memory-efficient Learning for High-Dimensional MRI Reconstruction
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• 8-bit Optimizers via Block-wise Quantization
• Fully reversible neural networks for large-scale surface and sub-surface characterization via remote sensing
• COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
• OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
• XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• Chest CT Super-resolution and Domain-adaptation using Memory-efficient 3D Reversible GANs
• Dynamic Tensor Rematerialization
• HybridNet: Classification and Reconstruction Cooperation for Semi-Supervised Learning
• SMYRF: Efficient Attention using Asymmetric Clustering
• Nonlinear Level Set Learning for Function Approximation on Sparse Data with Applications to Parametric Differential Equations
• Constrained Linear Data-feature Mapping for Image Classification
• Differentiate Everything with a Reversible Embeded Domain-Specific Language
• Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows
• Multi-Channel Volumetric Neural Network for Knee Cartilage Segmentation in Cone-beam CT
• Resampling Base Distributions of Normalizing Flows
• Going Deeper with Lean Point Networks
• Neural Parameter Allocation Search
• General Invertible Transformations for Flow-based Generative Modeling
• PointLIE: Locally Invertible Embedding for Point Cloud Sampling and Recovery
• Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs
• A Survey on Green Deep Learning
• Memory-efficient Learning for Large-scale Computational Imaging -- NeurIPS deep inverse workshop
• Implicit regularization of deep residual networks towards neural ODEs
• Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization