High-Dimensional Probability Estimation with Deep Density Models

Abstract

One of the fundamental problems in machine learning is the estimation of a probability distribution from data. Many techniques have been proposed to study the structure of data, most often building around the assumption that observations lie on a lower-dimensional manifold of high probability. It has been more difficult, however, to exploit this insight to build explicit, tractable density models for high-dimensional data. In this paper, we introduce the deep density model (DDM), a new approach to density estimation. We exploit insights from deep learning to construct a bijective map to a representation space, under which the transformation of the distribution of the data is approximately factorized and has identical and known marginal densities. The simplicity of the latent distribution under the model allows us to feasibly explore it, and the invertibility of the map to characterize contraction of measure across it. This enables us to compute normalized densities for out-of-sample data. This combination of tractability and flexibility allows us to tackle a variety of probabilistic tasks on high-dimensional datasets, including: rapid computation of normalized densities at test-time without evaluating a partition function; generation of samples without MCMC; and characterization of the joint entropy of the data.

12 pages, 4 figures, 1 table. Submitted for publication

Cited by in corpus (36)

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Normalizing Flows: An Introduction and Review of Current Methods
• Normalizing Flows for Probabilistic Modeling and Inference
• Masked Autoregressive Flow for Density Estimation
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Early Visual Concept Learning with Unsupervised Deep Learning
• Avoiding pathologies in very deep networks
• Discrete Flows: Invertible Generative Models of Discrete Data
• CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations
• Probabilistic Autoencoder
• Bayesian Stokes inversion with Normalizing flows
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Block Neural Autoregressive Flow
• Neural Approximate Sufficient Statistics for Implicit Models
• A RAD approach to deep mixture models
• Learning Discrete Distributions by Dequantization
• Invertible DenseNets with Concatenated LipSwish
• Integer Discrete Flows and Lossless Compression
• Information Losses in Neural Classifiers from Sampling
• E(n) Equivariant Normalizing Flows
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Approximating exponential family models (not single distributions) with a two-network architecture
• Improving Evolutionary Strategies with Generative Neural Networks
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• Sliced Iterative Normalizing Flows
• Identifying through Flows for Recovering Latent Representations
• Principled Interpolation in Normalizing Flows
• General Probabilistic Surface Optimization and Log Density Estimation
• Neural Discriminant Analysis for Deep Speaker Embedding
• Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
• Improving Supervised Phase Identification Through the Theory of Information Losses
• Self Normalizing Flows
• Testing for Typicality with Respect to an Ensemble of Learned Distributions
• Flexible Prior Distributions for Deep Generative Models