XCiT: Cross-Covariance Image Transformers

Abstract

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

References in corpus (25)

• Longformer: The Long-Document Transformer
• YFCC100M: The New Data in Multimedia Research
• Is Space-Time Attention All You Need for Video Understanding?
• MLP-Mixer: An all-MLP Architecture for Vision
• Transformer in Transformer
• Linformer: Self-Attention with Linear Complexity
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Particular object retrieval with integral max-pooling of CNN activations
• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Big Bird: Transformers for Longer Sequences
• High-Performance Large-Scale Image Recognition Without Normalization
• Perceiver: General Perception with Iterative Attention
• Training Vision Transformers for Image Retrieval
• Rethinking Attention with Performers
• Self-Supervised Learning with Swin Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Multiscale Vision Transformers
• MultiGrain: a unified image embedding for classes and instances
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• The iNaturalist Species Classification and Detection Dataset

Cited by in corpus (20)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Improving Chest X-Ray Report Generation by Leveraging Warm Starting
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
• Benchmarking Detection Transfer Learning with Vision Transformers
• Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention
• Prediction Calibration for Generalized Few-shot Semantic Segmentation
• A Convolutional Vision Transformer for Semantic Segmentation of Side-Scan Sonar Data
• Self-supervision for medical image classification: state-of-the-art performance with ~100 labeled training samples per class
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• Deep neural networks approach to microbial colony detection -- a comparative analysis
• DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning
• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• Cluster-based human-in-the-loop strategy for improving machine learning-based circulating tumor cell detection in liquid biopsy
• TransMix: Attend to Mix for Vision Transformers
• Long-range and hierarchical language predictions in brains and algorithms
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• Global Interaction Modelling in Vision Transformer via Super Tokens