Autoregressive Energy Machines

Abstract

Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.

Cited by in corpus (17)

• Deep Unsupervised Cardinality Estimation
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Neural Spline Flows
• NeuroCard: One Cardinality Estimator for All Tables
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• TraDE: Transformers for Density Estimation
• Invertible Generative Modeling using Linear Rational Splines
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• A Unified Transferable Model for ML-Enhanced DBMS
• Telescoping Density-Ratio Estimation
• Implicit Behavioral Cloning
• Autoregressive Score Matching
• Density Deconvolution with Normalizing Flows
• Arbitrary Conditional Distributions with Energy
• Autoencoding Under Normalization Constraints
• GFlowNet Foundations