PixelSNAIL: An Improved Autoregressive Generative Model

Abstract

Autoregressive generative models consistently achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions, which offer better access to earlier parts of the sequence than conventional RNNs. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this note, we describe the resulting model and present state-of-the-art log-likelihood results on CIFAR-10 (2.85 bits per dim) and $32\times 32$ ImageNet (3.80 bits per dim). Our implementation is available at https://github.com/neocxi/pixelsnail-public

References in corpus (1)

• Density estimation using Real NVP

Cited by in corpus (17)

• Generating Long Sequences with Sparse Transformers
• VideoGPT: Video Generation using VQ-VAE and Transformers
• MelNet: A Generative Model for Audio in the Frequency Domain
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Universal Planning Networks
• Autoregressive Quantile Networks for Generative Modeling
• Generating Subsurface Earth Models using Discrete Representation Learning and Deep Autoregressive Network
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Associative Compression Networks for Representation Learning
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• MaCow: Masked Convolutional Generative Flow
• Endowing Deep 3D Models with Rotation Invariance Based on Principal Component Analysis
• Hyperparameter optimization with REINFORCE and Transformers
• Resampling Base Distributions of Normalizing Flows
• RBM-Flow and D-Flow: Invertible Flows with Discrete Energy Base Spaces
• Generative Model with Dynamic Linear Flow