Generating Images with Sparse Representations

Abstract

The high dimensionality of images presents architecture and sampling-efficiency challenges for likelihood-based generative models. Previous approaches such as VQ-VAE use deep autoencoders to obtain compact representations, which are more practical as inputs for likelihood-based models. We present an alternative approach, inspired by common image compression methods like JPEG, and convert images to quantized discrete cosine transform (DCT) blocks, which are represented sparsely as a sequence of DCT channel, spatial location, and DCT coefficient triples. We propose a Transformer-based autoregressive architecture, which is trained to sequentially predict the conditional distribution of the next element in such sequences, and which scales effectively to high resolution images. On a range of image datasets, we demonstrate that our approach can generate high quality, diverse images, with sample metric scores competitive with state of the art methods. We additionally show that simple modifications to our method yield effective image colorization and super-resolution models.

References in corpus (13)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Discrete Representation Learning
• Pixel Recurrent Neural Networks
• Scaling Laws for Neural Language Models
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Variational Inference with Normalizing Flows
• Generating Long Sequences with Sparse Transformers
• Reformer: The Efficient Transformer
• ReZero is All You Need: Fast Convergence at Large Depth
• Jukebox: A Generative Model for Music
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling

Cited by in corpus (7)

• Diffusion Models Beat GANs on Image Synthesis
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Variable-rate discrete representation learning
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform
• NomMer: Nominate Synergistic Context in Vision Transformer for Visual Recognition
• PatchGame: Learning to Signal Mid-level Patches in Referential Games