Generative Adversarial Transformers

Abstract

We introduce the GANformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model's strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model's inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. An implementation of the model is available at https://github.com/dorarad/gansformer.

arXiv commentPublished as a conference paper at ICML 2021

References in corpus (24)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Adversarial Feature Learning
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• Improved Precision and Recall Metric for Assessing Generative Models
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• Multi-Object Representation Learning with Iterative Variational Inference
• Image Transformer
• Object-Centric Learning with Slot Attention
• MONet: Unsupervised Scene Decomposition and Representation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Recurrent Independent Mechanisms
• StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Investigating Object Compositionality in Generative Adversarial Networks
• ETC: Encoding Long and Structured Inputs in Transformers
• Global Self-Attention Networks for Image Recognition
• Generating unseen complex scenes: are we there yet?
• Semantic Bottleneck Scene Generation

Cited by in corpus (9)

• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer
• FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
• Styleformer: Transformer based Generative Adversarial Networks with Style Vector
• Bridging Global Context Interactions for High-Fidelity Image Completion
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Generating Symbolic Reasoning Problems with Transformer GANs
• Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?