Hierarchical Autoregressive Image Models with Auxiliary Decoders

Abstract

Autoregressive generative models of images tend to be biased towards capturing local structure, and as a result they often produce samples which are lacking in terms of large-scale coherence. To address this, we propose two methods to learn discrete representations of images which abstract away local detail. We show that autoregressive models conditioned on these representations can produce high-fidelity reconstructions of images, and that we can train autoregressive priors on these representations that produce samples with large-scale coherence. We can recursively apply the learning procedure, yielding a hierarchy of progressively more abstract image representations. We train hierarchical class-conditional autoregressive models on the ImageNet dataset and demonstrate that they are able to generate realistic images at resolutions of 128$\times$128 and 256$\times$256 pixels. We also perform a human evaluation study comparing our models with both adversarial and likelihood-based state-of-the-art generative models.

arXiv commentUpdated: added human evaluation results, incorporated review feedback

References in corpus (22)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Representation Learning with Contrastive Predictive Coding
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Discrete Representation Learning
• Conditional Image Generation with PixelCNN Decoders
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Variational Lossy Autoencoder
• A Note on the Inception Score
• The challenge of realistic music generation: modelling raw audio at scale
• Generative Image Modeling Using Spatial LSTMs
• PixelNet: Representation of the pixels, by the pixels, and for the pixels
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Z-Forcing: Training Stochastic Recurrent Networks
• Fast Wavenet Generation Algorithm

Cited by in corpus (14)

• Cascaded Diffusion Models for High Fidelity Image Generation
• Classification Accuracy Score for Conditional Generative Models
• Jukebox: A Generative Model for Music
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities
• Hierarchical Quantized Autoencoders
• Predicting Video with VQVAE
• Scaling Autoregressive Video Models
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• Variable-rate discrete representation learning
• Do sequence-to-sequence VAEs learn global features of sentences?
• not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
• Neural Communication Systems with Bandwidth-limited Channel