Globally Injective ReLU Networks

Abstract

Injectivity plays an important role in generative models where it enables inference; in inverse problems and compressed sensing with generative priors it is a precursor to well posedness. We establish sharp characterizations of injectivity of fully-connected and convolutional ReLU layers and networks. First, through a layerwise analysis, we show that an expansivity factor of two is necessary and sufficient for injectivity by constructing appropriate weight matrices. We show that global injectivity with iid Gaussian matrices, a commonly used tractable model, requires larger expansivity between 3.4 and 10.5. We also characterize the stability of inverting an injective network via worst-case Lipschitz constants of the inverse. We then use arguments from differential topology to study injectivity of deep networks and prove that any Lipschitz map can be approximated by an injective ReLU network. Finally, using an argument based on random projections, we show that an end-to-end -- rather than layerwise -- doubling of the dimension suffices for injectivity. Our results establish a theoretical basis for the study of nonlinear inverse and inference problems using neural networks.

arXiv comment48 pages, 18 figures, submitted to JMLR

References in corpus (22)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Adversarial Feature Learning
• Adversarially Learned Inference
• Compressed Sensing using Generative Models
• Neural Photo Editing with Introspective Adversarial Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• The Reversible Residual Network: Backpropagation Without Storing Activations
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Deep Compressed Sensing
• Is Generator Conditioning Causally Related to GAN Performance?
• Approximability of Discriminators Implies Diversity in GANs
• Signal Recovery from Pooling Representations
• Regularized Autoencoders via Relaxed Injective Probability Flow
• When and How Can Deep Generative Models be Inverted?
• Max-Affine Spline Insights into Deep Generative Networks
• Constant-Expansion Suffices for Compressed Sensing with Generative Priors

Cited by in corpus (5)

• Counterfactual Zero-Shot and Open-Set Visual Recognition
• Trumpets: Injective Flows for Inference and Inverse Problems
• Transporting Causal Mechanisms for Unsupervised Domain Adaptation
• Continuous Generative Neural Networks: A Wavelet-Based Architecture in Function Spaces
• Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows