RNADE: The real-valued neural autoregressive density-estimator

Abstract

We introduce RNADE, a new model for joint density estimation of real-valued vectors. Our model calculates the density of a datapoint as the product of one-dimensional conditionals modeled using mixture density networks with shared parameters. RNADE learns a distributed representation of the data, while having a tractable expression for the calculation of densities. A tractable likelihood allows direct comparison with other methods and training by standard gradient-based optimizers. We compare the performance of RNADE on several datasets of heterogeneous and perceptual data, finding it outperforms mixture models in all but one case.

12 pages, 3 figures, 3 tables, 2 algorithms. Merges the published paper and supplementary material into one document

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• Practical Bayesian Optimization of Machine Learning Algorithms
• No More Pesky Learning Rates
• Gaussian Process Networks
• Mixtures of conditional Gaussian scale mixtures applied to multiscale image representations
• Deep Mixtures of Factor Analysers
• In All Likelihood, Deep Belief Is Not Enough

Cited by in corpus (43)

• A note on the evaluation of generative models
• Normalizing Flows for Probabilistic Modeling and Inference
• Deep Learning-Based Video Coding: A Review and A Case Study
• Implicit Generation and Generalization in Energy-Based Models
• Generative Image Modeling Using Spatial LSTMs
• A Deep and Tractable Density Estimator
• Nonparametric Density Estimation for High-Dimensional Data - Algorithms and Applications
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds
• Neural Spline Flows
• SARM: Sparse Autoregressive Model for Scalable Generation of Sparse Images in Particle Physics
• Neural Autoregressive Distribution Estimation
• Neural Likelihoods via Cumulative Distribution Functions
• Categorical Normalizing Flows via Continuous Transformations
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Low-rank Characteristic Tensor Density Estimation Part I: Foundations
• Low-rank Characteristic Tensor Density Estimation Part II: Compression and Latent Density Estimation
• LogitBoost autoregressive networks
• Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
• Learning for Integer-Constrained Optimization through Neural Networks with Limited Training
• Locally Masked Convolution for Autoregressive Models
• Learning to Generate Genotypes with Neural Networks
• MaCow: Masked Convolutional Generative Flow
• Approximating exponential family models (not single distributions) with a two-network architecture
• Improved Autoregressive Modeling with Distribution Smoothing
• Best-scored Random Forest Density Estimation
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
• Likelihood Contribution based Multi-scale Architecture for Generative Flows
• General Probabilistic Surface Optimization and Log Density Estimation
• A Forest from the Trees: Generation through Neighborhoods
• Symmetric Wasserstein Autoencoders
• SECRET: Stochasticity Emulator for Cosmic Ray Electrons
• OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?
• Sampling in Combinatorial Spaces with SurVAE Flow Augmented MCMC
• Decoupling Global and Local Representations via Invertible Generative Flows
• Gradient Boosted Normalizing Flows
• Knothe-Rosenblatt transport for Unsupervised Domain Adaptation
• Imitation with Neural Density Models