Invertible Residual Networks

Abstract

We show that standard ResNet architectures can be made invertible, allowing the same model to be used for classification, density estimation, and generation. Typically, enforcing invertibility requires partitioning dimensions or restricting network architectures. In contrast, our approach only requires adding a simple normalization step during training, already available in standard frameworks. Invertible ResNets define a generative model which can be trained by maximum likelihood on unlabeled data. To compute likelihoods, we introduce a tractable approximation to the Jacobian log-determinant of a residual block. Our empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture.

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• Representation Learning with Contrastive Predictive Coding
• WaveNet: A Generative Model for Raw Audio
• Spectral Normalization for Generative Adversarial Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Wide Residual Networks
• Learning deep representations by mutual information estimation and maximization
• Pixel Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• MADE: Masked Autoencoder for Distribution Estimation
• Robust Large Margin Deep Neural Networks
• Towards the first adversarially robust neural network model on MNIST
• Image Transformer
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Analyzing Inverse Problems with Invertible Neural Networks
• The Singular Values of Convolutional Layers
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Sorting out Lipschitz function approximation
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Hybrid Models with Deep and Invertible Features
• Estimating the Spectral Density of Large Implicit Matrices
• Approximating the Spectral Sums of Large-scale Matrices using Chebyshev Approximations
• Stochastic Chebyshev Gradient Descent for Spectral Optimization
• Backpropagation for Implicit Spectral Densities

Cited by in corpus (51)

• Normalizing Flows: An Introduction and Review of Current Methods
• Image Quality Assessment: Unifying Structure and Texture Similarity
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Time-series learning of latent-space dynamics for reduced-order model closure
• Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities
• Revisiting Self-Supervised Visual Representation Learning
• Excessive Invariance Causes Adversarial Vulnerability
• Augmented Neural ODEs
• Robust in Practice: Adversarial Attacks on Quantum Machine Learning
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• InversionNet3D: Efficient and Scalable Learning for 3D Full Waveform Inversion
• Why Normalizing Flows Fail to Detect Out-of-Distribution Data
• Invertible Network for Classification and Biomarker Selection for ASD
• An introduction to variational inference in Geophysical inverse problems
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• Benchmarking Invertible Architectures on Inverse Problems
• Block Neural Autoregressive Flow
• Stochastic Normalizing Flows
• $m^{\ast }$ of two-dimensional electron gas: a neural canonical transformation study
• Soft-Root-Sign Activation Function
• Tail of Distribution GAN (TailGAN): Generative-Adversarial-Network-Based Boundary Formation
• Training Invertible Neural Networks as Autoencoders
• Boundary of Distribution Support Generator (BDSG): Sample Generation on the Boundary
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Robust and Provably Monotonic Networks
• Flow Models for Arbitrary Conditional Likelihoods
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• Enhancing Image Rescaling using Dual Latent Variables in Invertible Neural Network
• Neural Density Estimation and Likelihood-free Inference
• Hybrid Models for Open Set Recognition
• Semi-Supervised Learning with Normalizing Flows
• Stochastic Neural Network with Kronecker Flow
• Invertible residual networks in the context of regularization theory for linear inverse problems
• Generative Machine Learning for Multivariate Equity Returns
• A Shooting Formulation of Deep Learning
• A Plug-in Method for Representation Factorization in Connectionist Models
• HyperFlow: Representing 3D Objects as Surfaces
• Learning nonlinear level sets for dimensionality reduction in function approximation
• Efficient Realistic Data Generation Framework leveraging Deep Learning-based Human Digitization
• Exponential Tilting of Generative Models: Improving Sample Quality by Training and Sampling from Latent Energy
• Differential equations as models of deep neural networks
• FIS-GAN: GAN with Flow-based Importance Sampling
• Getting Topology and Point Cloud Generation to Mesh
• Woodbury Transformations for Deep Generative Flows
• Taming Reversible Halftoning via Predictive Luminance
• Differentiate Everything with a Reversible Embeded Domain-Specific Language
• Memory-efficient Learning for Large-scale Computational Imaging -- NeurIPS deep inverse workshop
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More