Parallel Multiscale Autoregressive Density Estimation

Abstract

PixelCNN achieves state-of-the-art results in density estimation for natural images. Although training is fast, inference is costly, requiring one network evaluation per pixel; O(N) for N pixels. This can be sped up by caching activations, but still involves generating each pixel sequentially. In this work, we propose a parallelized PixelCNN that allows more efficient inference by modeling certain pixel groups as conditionally independent. Our new PixelCNN model achieves competitive density estimation and orders of magnitude speedup - O(log N) sampling instead of O(N) - enabling the practical generation of 512x512 images. We evaluate the model on class-conditional image generation, text-to-image synthesis, and action-conditional video generation, showing that our model achieves the best results among non-pixel-autoregressive density models that allow efficient sampling.

References in corpus (8)

• Generative Adversarial Text to Image Synthesis
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Density estimation using Real NVP
• Neural Machine Translation in Linear Time
• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
• Learning What and Where to Draw
• Generative Image Modeling Using Spatial LSTMs

Cited by in corpus (19)

• Generating Long Sequences with Sparse Transformers
• Joint Autoregressive and Hierarchical Priors for Learned Image Compression
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Adversarial Text-to-Image Synthesis: A Review
• MelNet: A Generative Model for Audio in the Frequency Domain
• Self-Supervised Visual Planning with Temporal Skip Connections
• Theory and Experiments on Vector Quantized Autoencoders
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
• Pixel Deconvolutional Networks
• Lossless Image Compression through Super-Resolution
• NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
• HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models
• CAGAN: Text-To-Image Generation with Combined Attention GANs
• Image Generation from Layout
• PixelCNN Models with Auxiliary Variables for Natural Image Modeling
• SegAttnGAN: Text to Image Generation with Segmentation Attention
• Neural Multi-scale Image Compression
• Sketch-Inspector: a Deep Mixture Model for High-Quality Sketch Generation of Cats