TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

Abstract

The recent explosive interest on transformers has suggested their potential to become powerful "universal" models for computer vision tasks, such as classification, detection, and segmentation. While those attempts mainly study the discriminative models, we explore transformers on some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs). Our goal is to conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures. On top of them, we introduce the new module of grid self-attention for alleviating the memory bottleneck further, in order to scale up TransGAN to high-resolution generation. We also develop a unique training recipe including a series of techniques that can mitigate the training instability issues of TransGAN, such as data augmentation, modified normalization, and relative position encoding. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs using convolutional backbones. Specifically, TransGAN sets new state-of-the-art inception score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. When it comes to higher-resolution (e.g. 256 x 256) generation tasks, such as on CelebA-HQ and LSUN-Church, TransGAN continues to produce diverse visual examples with high fidelity and impressive texture details. In addition, we dive deep into the transformer-based generation models to understand how their behaviors differ from convolutional ones, by visualizing training dynamics. The code is available at https://github.com/VITA-Group/TransGAN.

Accepted to NeruIPS 2021

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Is Space-Time Attention All You Need for Video Understanding?
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Dual Discriminator Generative Adversarial Nets
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• Consistency Regularization for Generative Adversarial Networks
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Generative Adversarial Transformers

Cited by in corpus (23)

• A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization
• Uformer: A General U-Shaped Transformer for Image Restoration
• One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• ViTGAN: Training GANs with Vision Transformers
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• A Transformer-based Generative Adversarial Network for Brain Tumor Segmentation
• PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion
• Vision Transformers with Patch Diversification
• A Survey of Visual Transformers
• Delving Globally into Texture and Structure for Image Inpainting
• Generative Adversarial Transformers
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning
• SDNet: mutil-branch for single image deraining using swin
• Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition
• FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• A Mathematical Framework for Learning Probability Distributions
• Improved Transformer for High-Resolution GANs
• Generating Symbolic Reasoning Problems with Transformer GANs
• Pathologies in priors and inference for Bayesian transformers