Generative Adversarial Transformers

Abstract

We introduce the GANformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model's strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model's inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. An implementation of the model is available at https://github.com/dorarad/gansformer.

Published as a conference paper at ICML 2021

References in corpus (8)

• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• MONet: Unsupervised Scene Decomposition and Representation
• StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation
• Global Self-Attention Networks for Image Recognition
• Generating unseen complex scenes: are we there yet?
• Semantic Bottleneck Scene Generation

Cited by in corpus (9)

• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer
• FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
• Styleformer: Transformer based Generative Adversarial Networks with Style Vector
• Bridging Global Context Interactions for High-Fidelity Image Completion
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Generating Symbolic Reasoning Problems with Transformer GANs
• Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?