Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Abstract

Whilst diffusion probabilistic models can generate high quality image content, key limitations remain in terms of both generating high-resolution imagery and their associated high computational requirements. Recent Vector-Quantized image models have overcome this limitation of image resolution but are prohibitively slow and unidirectional as they generate tokens via element-wise autoregressive sampling from the prior. By contrast, in this paper we propose a novel discrete diffusion probabilistic model prior which enables parallel prediction of Vector-Quantized tokens by using an unconstrained Transformer architecture as the backbone. During training, tokens are randomly masked in an order-agnostic manner and the Transformer learns to predict the original tokens. This parallelism of Vector-Quantized token prediction in turn facilitates unconditional generation of globally consistent high-resolution and diverse imagery at a fraction of the computational expense. In this manner, we can generate image resolutions exceeding that of the original training set samples whilst additionally provisioning per-image likelihood estimates (in a departure from generative adversarial approaches). Our approach achieves state-of-the-art results in terms of Density (LSUN Bedroom: 1.51; LSUN Churches: 1.12; FFHQ: 1.20) and Coverage (LSUN Bedroom: 0.83; LSUN Churches: 0.73; FFHQ: 0.80), and performs competitively on FID (LSUN Bedroom: 3.64; LSUN Churches: 4.07; FFHQ: 6.11) whilst offering advantages in terms of both computation and reduced training set requirements.

19 pages, 14 figures

References in corpus (16)

• Learning Transferable Visual Models From Natural Language Supervision
• Zero-Shot Text-to-Image Generation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Generating Long Sequences with Sparse Transformers
• Cascaded Diffusion Models for High Fidelity Image Generation
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Variational Diffusion Models
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis
• Structured Denoising Diffusion Models in Discrete State-Spaces
• UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Learning to Efficiently Sample from Diffusion Probabilistic Models
• Generating Images with Sparse Representations
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds