Understanding and Mitigating Exploding Inverses in Invertible Neural Networks

Abstract

Invertible neural networks (INNs) have been used to design generative models, implement memory-saving gradient computation, and solve inverse problems. In this work, we show that commonly-used INN architectures suffer from exploding inverses and are thus prone to becoming numerically non-invertible. Across a wide range of INN use-cases, we reveal failures including the non-applicability of the change-of-variables formula on in- and out-of-distribution (OOD) data, incorrect gradients for memory-saving backprop, and the inability to sample from normalizing flow models. We further derive bi-Lipschitz properties of atomic building blocks of common architectures. These insights into the stability of INNs then provide ways forward to remedy these failures. For tasks where local invertibility is sufficient, like memory-saving backprop, we propose a flexible and efficient regularizer. For problems where global invertibility is necessary, such as applying normalizing flows on OOD data, we show the importance of designing stable INN building blocks.

AISTATS 2021

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• NICE: Non-linear Independent Components Estimation
• Guided Image Generation with Conditional Invertible Neural Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Improving Variational Auto-Encoders using Householder Flow
• Sum-of-Squares Polynomial Flow
• Invert to Learn to Invert
• iUNets: Fully invertible U-Nets with Learnable Up- and Downsampling
• A Closer Look at Double Backpropagation

Cited by in corpus (10)

• Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Stabilizing Invertible Neural Networks Using Mixture Models
• Perfect density models cannot guarantee anomaly detection
• Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization
• Flow-based Self-supervised Density Estimation for Anomalous Sound Detection
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression
• The Effects of Invertibility on the Representational Complexity of Encoders in Variational Autoencoders
• Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization