TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

Abstract

The recent explosive interest on transformers has suggested their potential to become powerful "universal" models for computer vision tasks, such as classification, detection, and segmentation. While those attempts mainly study the discriminative models, we explore transformers on some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs). Our goal is to conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures. On top of them, we introduce the new module of grid self-attention for alleviating the memory bottleneck further, in order to scale up TransGAN to high-resolution generation. We also develop a unique training recipe including a series of techniques that can mitigate the training instability issues of TransGAN, such as data augmentation, modified normalization, and relative position encoding. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs using convolutional backbones. Specifically, TransGAN sets new state-of-the-art inception score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. When it comes to higher-resolution (e.g. 256 x 256) generation tasks, such as on CelebA-HQ and LSUN-Church, TransGAN continues to produce diverse visual examples with high fidelity and impressive texture details. In addition, we dive deep into the transformer-based generation models to understand how their behaviors differ from convolutional ones, by visualizing training dynamics. The code is available at https://github.com/VITA-Group/TransGAN.

arXiv commentAccepted to NeruIPS 2021

References in corpus (36)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Self-Attention Generative Adversarial Networks
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Is Space-Time Attention All You Need for Video Understanding?
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Toward Multimodal Image-to-Image Translation
• Training Generative Adversarial Networks with Limited Data
• Linformer: Self-Attention with Linear Complexity
• Which Training Methods for GANs do actually Converge?
• The relativistic discriminator: a key element missing from standard GAN
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Generating Long Sequences with Sparse Transformers
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• Pre-Trained Image Processing Transformer
• Dual Discriminator Generative Adversarial Nets
• Rethinking Attention with Performers
• Differentiable Augmentation for Data-Efficient GAN Training
• Pedestrian-Synthesis-GAN: Generating Pedestrian Data in Real Scene and Beyond
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• Consistency Regularization for Generative Adversarial Networks
• Class-Splitting Generative Adversarial Networks
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Generative Adversarial Transformers
• Distilling portable Generative Adversarial Networks for Image Translation

Cited by in corpus (24)

• A Survey on Visual Transformer
• A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization
• Uformer: A General U-Shaped Transformer for Image Restoration
• One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• ViTGAN: Training GANs with Vision Transformers
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• A Transformer-based Generative Adversarial Network for Brain Tumor Segmentation
• PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion
• A Survey of Visual Transformers
• Vision Transformers with Patch Diversification
• Delving Globally into Texture and Structure for Image Inpainting
• Generative Adversarial Transformers
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning
• SDNet: mutil-branch for single image deraining using swin
• Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition
• FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• A Mathematical Framework for Learning Probability Distributions
• Improved Transformer for High-Resolution GANs
• Pathologies in priors and inference for Bayesian transformers
• Generating Symbolic Reasoning Problems with Transformer GANs