Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators

Abstract

Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. However, their desirable characteristics such as analytic invertibility come at the cost of restricting the functional forms. This poses a question on their representation power: are CF-INNs universal approximators for invertible functions? Without a universality, there could be a well-behaved invertible transformation that the CF-INN can never approximate, hence it would render the model class unreliable. We answer this question by showing a convenient criterion: a CF-INN is universal if its layers contain affine coupling and invertible linear functions as special cases. As its corollary, we can affirmatively resolve a previously unsolved problem: whether normalizing flow models based on affine coupling can be universal distributional approximators. In the course of proving the universality, we prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes, a theoretical insight that is of interest by itself.

29 pages, 3 figures. Accepted at Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020) for oral presentation

References in corpus (13)

• NICE: Non-linear Independent Components Estimation
• Normalizing Flows: An Introduction and Review of Current Methods
• Density estimation using Real NVP
• Neural Ordinary Differential Equations
• Normalizing Flows for Probabilistic Modeling and Inference
• Analyzing Inverse Problems with Invertible Neural Networks
• ResNet with one-neuron hidden layers is a Universal Approximator
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• Augmented Neural ODEs
• Few-shot Domain Adaptation by Causal Mechanism Transfer
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• Density Matching for Bilingual Word Embedding

Cited by in corpus (16)

• On the representation and learning of monotone triangular transport maps
• PatchNR: Learning from Very Few Images by Patch Normalizing Flow Regularization
• Neural Information Squeezer for Causal Emergence
• Bridging Mean-Field Games and Normalizing Flows with Trajectory Regularization
• Memory-Efficient Network for Large-scale Video Compressive Sensing
• Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization
• Trumpets: Injective Flows for Inference and Inverse Problems
• Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification
• Representational aspects of depth and conditioning in normalizing flows
• Generative Machine Learning for Multivariate Equity Returns
• Neural Spatio-Temporal Point Processes
• Sparse Flows: Pruning Continuous-depth Models
• Variational Monte Carlo Approach to Partial Differential Equations with Neural Networks
• Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows
• EMFlow: Data Imputation in Latent Space via EM and Deep Flow Models
• On the Generative Utility of Cyclic Conditionals