i-RevNet: Deep Invertible Networks

Abstract

It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

References in corpus (6)

• NICE: Non-linear Independent Components Estimation
• Opening the Black Box of Deep Neural Networks via Information
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Parseval Networks: Improving Robustness to Adversarial Examples
• Understanding deep features with computer-generated imagery
• Building a Regular Decision Boundary with Deep Networks

Cited by in corpus (34)

• Hamiltonian Neural Networks
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• DIVA: Domain Invariant Variational Autoencoders
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Invertible generative models for inverse problems: mitigating representation error and dataset bias
• Invertible Network for Classification and Biomarker Selection for ASD
• Explaining Knowledge Distillation by Quantifying the Knowledge
• iUNets: Fully invertible U-Nets with Learnable Up- and Downsampling
• Training Invertible Neural Networks as Autoencoders
• Latent Space Factorisation and Manipulation via Matrix Subspace Projection
• ChartPointFlow for Topology-Aware 3D Point Cloud Generation
• Training Generative Reversible Networks
• GLOWin: A Flow-based Invertible Generative Framework for Learning Disentangled Feature Representations in Medical Images
• CrevNet: Conditionally Reversible Video Prediction
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks
• Estimating Differential Entropy under Gaussian Convolutions
• Trumpets: Injective Flows for Inference and Inverse Problems
• Approximating exponential family models (not single distributions) with a two-network architecture
• A Differential Topological View of Challenges in Learning with Feedforward Neural Networks
• Conditional Normalizing Flows for Low-Dose Computed Tomography Image Reconstruction
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• Unsupervised Learning of Syntactic Structure with Invertible Neural Projections
• Learning nonlinear level sets for dimensionality reduction in function approximation
• Featurized Density Ratio Estimation
• Decision Explanation and Feature Importance for Invertible Networks
• Towards Learning a Self-inverse Network for Bidirectional Image-to-image Translation
• One-to-one Mapping for Unpaired Image-to-image Translation
• Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows
• Input Invex Neural Network
• Data-driven Regularization via Racecar Training for Generalizing Neural Networks
• PointLIE: Locally Invertible Embedding for Point Cloud Sampling and Recovery