Transformer in Transformer

Abstract

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.

arXiv commentAccepted by NeurIPS 2021

References in corpus (10)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• mixup: Beyond Empirical Risk Minimization
• A Survey on Visual Transformer
• Gaussian Error Linear Units (GELUs)
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• FractalNet: Ultra-Deep Neural Networks without Residuals
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Toward Transformer-Based Object Detection
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Cited by in corpus (81)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Attention Mechanisms in Computer Vision: A Survey
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• LocalViT: Analyzing Locality in Vision Transformers
• CTformer: Convolution-free Token2Token Dilated Vision Transformer for Low-dose CT Denoising
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• XCiT: Cross-Covariance Image Transformers
• CvT: Introducing Convolutions to Vision Transformers
• TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Mapping Degeneration Meets Label Evolution: Learning Infrared Small Target Detection with Single Point Supervision
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• DPT: Deformable Patch-based Transformer for Visual Recognition
• Point Cloud Classification Using Content-based Transformer via Clustering in Feature Space
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• RegionViT: Regional-to-Local Attention for Vision Transformers
• AS-MLP: An Axial Shifted MLP Architecture for Vision
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• ViTA: A Vision Transformer Inference Accelerator for Edge Applications
• CNN-LSTM and Transfer Learning Models for Malware Classification based on Opcodes and API Calls
• CMT: Convolutional Neural Networks Meet Vision Transformers
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• Application of belief functions to medical image segmentation: A review
• Face Transformer for Recognition
• Vision Transformers with Patch Diversification
• A Survey of Visual Transformers
• Refiner: Refining Self-attention for Vision Transformers
• Dual-stream Network for Visual Recognition
• Glance-and-Gaze Vision Transformer
• UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
• Fully Transformer Networks for Semantic Image Segmentation
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• MetaFormer Is Actually What You Need for Vision
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• MCI-GRU: Stock Prediction Model Based on Multi-Head Cross-Attention and Improved GRU
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM
• ElegansNet: a brief scientific report and initial experiments
• VOLO: Vision Outlooker for Visual Recognition
• MVT: Multi-view Vision Transformer for 3D Object Recognition
• A Full Transformer-based Framework for Automatic Pain Estimation using Videos
• DuDoTrans: Dual-Domain Transformer Provides More Attention for Sinogram Restoration in Sparse-View CT Reconstruction
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation
• Visual Parser: Representing Part-whole Hierarchies with Transformers
• Hierarchical Interdisciplinary Topic Detection Model for Research Proposal Classification
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• SpecTNT: a Time-Frequency Transformer for Music Audio
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• Searching for Efficient Multi-Stage Vision Transformers
• Brain Stroke Detection and Classification Using CT Imaging with Transformer Models and Explainable AI
• Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi
• Shunted Self-Attention via Multi-Scale Token Aggregation
• MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
• TransLoc3D : Point Cloud based Large-scale Place Recognition using Adaptive Receptive Fields
• Towards Robust Vision Transformer
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• COVID-19 Detection in Chest X-ray Images Using Swin-Transformer and Transformer in Transformer
• gSwin: Gated MLP Vision Model with Hierarchical Structure of Shifted Window
• Hire-MLP: Vision MLP via Hierarchical Rearrangement
• More than Encoder: Introducing Transformer Decoder to Upsample
• An Image Patch is a Wave: Phase-Aware Vision MLP
• KVT: k-NN Attention for Boosting Vision Transformers
• Boosting Salient Object Detection with Transformer-based Asymmetric Bilateral U-Net
• Post-Training Quantization for Vision Transformer
• Prioritized Architecture Sampling with Monto-Carlo Tree Search
• PointMixer: MLP-Mixer for Point Cloud Understanding
• ViTAS: Vision Transformer Architecture Search
• Gaze Estimation using Transformer
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• Global Interaction Modelling in Vision Transformer via Super Tokens