Predictive Sampling with Forecasting Autoregressive Models

Abstract

Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.

Accepted at the 37th International Conference on Machine Learning (ICML 2020). 14 pages, 13 figures

References in corpus (13)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• End-to-end Optimized Image Compression
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Generating Long Sequences with Sparse Transformers
• MADE: Masked Autoencoder for Distribution Estimation
• Video Compression With Rate-Distortion Autoencoders
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Spatial PixelCNN: Generating Images from Patches

Cited by in corpus (3)

• Denoising Diffusion Probabilistic Models
• Locally Masked Convolution for Autoregressive Models
• The Convolution Exponential and Generalized Sylvester Flows