NICE: Non-linear Independent Components Estimation

Abstract

We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is one in which the data has a distribution that is easy to model. For this purpose, a non-linear deterministic transformation of the data is learned that maps it to a latent space so as to make the transformed data conform to a factorized distribution, i.e., resulting in independent latent variables. We parametrize this transformation so that computing the Jacobian determinant and inverse transform is trivial, yet we maintain the ability to learn complex non-linear transformations, via a composition of simple building blocks, each based on a deep neural network. The training criterion is simply the exact log-likelihood, which is tractable. Unbiased ancestral sampling is also easy. We show that this approach yields good generative models on four image datasets and can be used for inpainting.

11 pages and 2 pages Appendix, workshop paper at ICLR 2015

References in corpus (4)

• Better Mixing via Deep Representations
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• High-Dimensional Probability Estimation with Deep Density Models
• Deep Directed Generative Autoencoders

Cited by in corpus (58)

• Guided Image Generation with Conditional Invertible Neural Networks
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Improving Variational Auto-Encoders using Householder Flow
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Learning Independent Features with Adversarial Nets for Non-linear ICA
• Independently Controllable Factors
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• Discrete Flows: Invertible Generative Models of Discrete Data
• Graph Normalizing Flows
• Generative Adversarial Parallelization
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Block Neural Autoregressive Flow
• Learning Symmetries of Classical Integrable Systems
• Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
• Style Generator Inversion for Image Enhancement and Animation
• Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flow
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Generation of ice states through deep reinforcement learning
• Neural Entropic Estimation: A faster path to mutual information estimation
• How to Hallucinate Functional Proteins
• Understanding the (un)interpretability of natural image distributions using generative models
• Supervised Anomaly Detection based on Deep Autoregressive Density Estimators
• Semi-Supervised Learning with Normalizing Flows
• CrevNet: Conditionally Reversible Video Prediction
• On Scalable and Efficient Computation of Large Scale Optimal Transport
• CMTS: Conditional Multiple Trajectory Synthesizer for Generating Safety-critical Driving Scenarios
• Multimodal Generative Models for Compositional Representation Learning
• Max-Affine Spline Insights into Deep Generative Networks
• A Method to Model Conditional Distributions with Normalizing Flows
• Generalized Zero-Shot Learning via VAE-Conditioned Generative Flow
• Improving Evolutionary Strategies with Generative Neural Networks
• Unpaired Image Denoising
• Continuous Graph Flow
• Learning nonlinear level sets for dimensionality reduction in function approximation
• Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
• FaceShapeGene: A Disentangled Shape Representation for Flexible Face Image Editing
• Point Process Flows
• Secure Data Sharing With Flow Model
• Deep Generative Models Strike Back! Improving Understanding and Evaluation in Light of Unmet Expectations for OoD Data
• Video Extrapolation with an Invertible Linear Embedding
• MIM: Mutual Information Machine
• GSNs : Generative Stochastic Networks
• Likelihood Assignment for Out-of-Distribution Inputs in Deep Generative Models is Sensitive to Prior Distribution Choice
• AE-OT-GAN: Training GANs from data specific latent distribution
• On Investigation of Unsupervised Speech Factorization Based on Normalization Flow
• Recurrent Estimation of Distributions
• One Reflection Suffice
• Quasi-Autoregressive Residual (QuAR) Flows
• TzK: Flow-Based Conditional Generative Model
• Generative Model with Dynamic Linear Flow
• Representation Learning: A Statistical Perspective
• Generative Parameter Sampler For Scalable Uncertainty Quantification
• One-element Batch Training by Moving Window