Autoregressive Diffusion Models
arXiv:2110.02037
Abstract
We introduce Autoregressive Diffusion Models (ARDMs), a model class encompassing and generalizing order-agnostic autoregressive models (Uria et al., 2014) and absorbing discrete diffusion (Austin et al., 2021), which we show are special cases of ARDMs under mild assumptions. ARDMs are simple to implement and easy to train. Unlike standard ARMs, they do not require causal masking of model representations, and can be trained using an efficient objective similar to modern probabilistic diffusion models that scales favourably to highly-dimensional data. At test time, ARDMs support parallel generation which can be adapted to fit any given generation budget. We find that ARDMs require significantly fewer steps than discrete diffusion models to attain the same performance. Finally, we apply ARDMs to lossless compression, and show that they are uniquely suited to this task. Contrary to existing approaches based on bits-back coding, ARDMs obtain compelling results not only on complete datasets, but also on compressing single data points. Moreover, this can be done using a modest number of network calls for (de)compression due to the model's adaptable parallel generation.
arXiv commentPublished as a conference paper at International Conference on Learning Representations (ICLR) 2022
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- WaveNet: A Generative Model for Raw Audio
- Diffusion Models Beat GANs on Image Synthesis
- Pixel Recurrent Neural Networks
- Generative Modeling by Estimating Gradients of the Data Distribution
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
- Generating Long Sequences with Sparse Transformers
- Variational Diffusion Models
- A Deep and Tractable Density Estimator
- Structured Denoising Diffusion Models in Discrete State-Spaces
- Asymmetric numeral systems
- Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
- On Fast Sampling of Diffusion Probabilistic Models
- Learning to Efficiently Sample from Diffusion Probabilistic Models
- Practical Lossless Compression with Latent Variables using Bits Back Coding
- A Variational Perspective on Diffusion-Based Generative Models and Score Matching
- Compression with Flows via Local Bits-Back Coding
- Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
- Gotta Go Fast When Generating Data with Score-Based Models
- Consistency Regularization for Variational Auto-Encoders
- Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving
- The DEformer: An Order-Agnostic Distribution Estimating Transformer