Image Transformer

Abstract

Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we find that images generated by our super-resolution model fool human observers three times more often than the previous state of the art.

arXiv commentAppears in International Conference on Machine Learning, 2018. Code available at https://github.com/tensorflow/tensor2tensor

Cited by in corpus (61)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Is Space-Time Attention All You Need for Video Understanding?
• A survey of the Vision Transformers and their CNN-Transformer based Variants
• Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
• NVAE: A Deep Hierarchical Variational Autoencoder
• Variational Diffusion Models
• Adversarial Text-to-Image Synthesis: A Review
• Invertible Residual Networks
• Rethinking Attention with Performers
• Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
• Deep Bayesian Inversion
• Copyright in Generative Deep Learning
• Bayesian Layers: A Module for Neural Network Uncertainty
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Full Page Handwriting Recognition via Image to Sequence Extraction
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows
• Long-Short Transformer: Efficient Transformers for Language and Vision
• An Image is Worth 16x16 Words, What is a Video Worth?
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Probabilistic Simulation of Quantum Circuits with the Transformer
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• Variational Transformers for Diverse Response Generation
• Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
• Creativity and Machine Learning: A Survey
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Simple, Distributed, and Accelerated Probabilistic Programming
• Few-Shot Segmentation via Cycle-Consistent Transformer
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
• Understanding the Difficulty of Training Transformers
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Perfect density models cannot guarantee anomaly detection
• QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
• Attention-based Dropout Layer for Weakly Supervised Object Localization
• Doubly Attentive Transformer Machine Translation
• Generative Adversarial Transformers
• Learning to Focus: Cascaded Feature Matching Network for Few-shot Image Recognition
• Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference
• Progressive Transformers for End-to-End Sign Language Production
• Scaling Autoregressive Video Models
• TraDE: Transformers for Density Estimation
• Faster Transformer Decoding: N-gram Masked Self-Attention
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Reformulating HOI Detection as Adaptive Set Prediction
• On the Integration of Self-Attention and Convolution
• Variational Inference for Computational Imaging Inverse Problems
• Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers
• SAFL: A Self-Attention Scene Text Recognizer with Focal Loss
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• Improved Transformer for High-Resolution GANs
• Machine Learning in Gamma Astronomy
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
• I2C2W: Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition
• Resampling Base Distributions of Normalizing Flows
• Token Pooling in Vision Transformers
• HyperHyperNetworks for the Design of Antenna Arrays