Improved Autoregressive Modeling with Distribution Smoothing
arXiv:2103.15089
Abstract
While autoregressive models excel at image compression, their sample quality is often lacking. Although not realistic, generated images often have high likelihood according to the model, resembling the case of adversarial examples. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling. We first model a smoothed version of the data distribution, and then reverse the smoothing process to recover the original data distribution. This procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world image datasets while obtaining competitive likelihoods on synthetic datasets.
arXiv commentICLR 2021 (Oral)
References in corpus (17)
- Denoising Diffusion Probabilistic Models
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Variational Inference with Normalizing Flows
- Density estimation using Real NVP
- Certified Adversarial Robustness via Randomized Smoothing
- Joint Autoregressive and Hierarchical Priors for Learned Image Compression
- MADE: Masked Autoencoder for Distribution Estimation
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Deep Generative Stochastic Networks Trainable by Backprop
- Improved Techniques for Training Score-Based Generative Models
- Implicit Generation and Generalization in Energy-Based Models
- RNADE: The real-valued neural autoregressive density-estimator
- Large sample analysis of the median heuristic
- Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
- Deep Energy Estimator Networks
- Autoregressive Quantile Networks for Generative Modeling
- Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows