Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling

Abstract

The unconditional generation of high fidelity images is a longstanding benchmark for testing the performance of image decoders. Autoregressive image models have been able to generate small images unconditionally, but the extension of these methods to large images where fidelity can be more readily assessed has remained an open problem. Among the major challenges are the capacity to encode the vast previous context and the sheer difficulty of learning a distribution that preserves both global semantic coherence and exactness of detail. To address the former challenge, we propose the Subscale Pixel Network (SPN), a conditional decoder architecture that generates an image as a sequence of sub-images of equal size. The SPN compactly captures image-wide spatial dependencies and requires a fraction of the memory and the computation required by other fully autoregressive models. To address the latter challenge, we propose to use Multidimensional Upscaling to grow an image in both size and depth via intermediate stages utilising distinct SPNs. We evaluate SPNs on the unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32 to 256. We achieve state-of-the-art likelihood results in multiple settings, set up new benchmark results in previously unexplored settings and are able to generate very high fidelity large scale samples on the basis of both datasets.

Cited by in corpus (37)

• Denoising Diffusion Probabilistic Models
• Evaluating Large Language Models Trained on Code
• Cascaded Diffusion Models for High Fidelity Image Generation
• Improved Denoising Diffusion Probabilistic Models
• NVAE: A Deep Hierarchical Variational Autoencoder
• Adversarial Video Generation on Complex Datasets
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• High-Fidelity Image Generation With Fewer Labels
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• PolyGen: An Autoregressive Generative Model of 3D Meshes
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• A Large-Scale Study on Regularization and Normalization in GANs
• Combiner: Full Attention Transformer with Sparse Computation Cost
• Predicting Video with VQVAE
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• One Epoch Is All You Need
• Image Generation From Small Datasets via Batch Statistics Adaptation
• Scaling Autoregressive Video Models
• Score-based Generative Modeling in Latent Space
• Densely connected normalizing flows
• Variable-rate discrete representation learning
• HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models
• Generating Images with Sparse Representations
• Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• MaCow: Masked Convolutional Generative Flow
• Improved Autoregressive Modeling with Distribution Smoothing
• Blur, Noise, and Compression Robust Generative Adversarial Networks
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
• not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
• Decoupling Global and Local Representations via Invertible Generative Flows
• Autoregressive Diffusion Models
• Normalizing Flows with Multi-Scale Autoregressive Priors
• Interpreting Spatially Infinite Generative Models
• Noise Robust Generative Adversarial Networks
• PixelPyramids: Exact Inference Models from Lossless Image Pyramids