TraDE: Transformers for Density Estimation

Abstract

We present TraDE, a self-attention-based architecture for auto-regressive density estimation with continuous and discrete valued data. Our model is trained using a penalized maximum likelihood objective, which ensures that samples from the density estimate resemble the training data distribution. The use of self-attention means that the model need not retain conditional sufficient statistics during the auto-regressive process beyond what is needed for each covariate. On standard tabular and image data benchmarks, TraDE produces significantly better density estimates than existing approaches such as normalizing flow estimators and recurrent auto-regressive models. However log-likelihood on held-out data only partially reflects how useful these estimates are in real-world applications. In order to systematically evaluate density estimators, we present a suite of tasks such as regression using generated samples, out-of-distribution detection, and robustness to noise in the training data and demonstrate that TraDE works well in these scenarios.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Categorical Reparameterization with Gumbel-Softmax
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• NICE: Non-linear Independent Components Estimation
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Density estimation using Real NVP
• Variational Dropout and the Local Reparameterization Trick
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• A note on the evaluation of generative models
• Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
• MADE: Masked Autoencoder for Distribution Estimation
• Image Transformer
• Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Sylvester Normalizing Flows for Variational Inference
• A Deep and Tractable Density Estimator
• Capacity and Trainability in Recurrent Neural Networks
• Language Models with Transformers
• Are Transformers universal approximators of sequence-to-sequence functions?
• On the optimal estimation of probability measures in weak and strong topologies
• Block Neural Autoregressive Flow
• Cubic-Spline Flows
• Autoregressive Energy Machines
• Transformation Autoregressive Networks

Cited by in corpus (8)

• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• Arbitrary Conditional Distributions with Energy
• The DEformer: An Order-Agnostic Distribution Estimating Transformer
• Sum-Product-Attention Networks: Leveraging Self-Attention in Probabilistic Circuits