Multiscale Vision Transformers

Abstract

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10x more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast

Technical report

References in corpus (35)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Learning Transferable Visual Models From Natural Language Supervision
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Res2Net: A New Multi-scale Backbone Architecture
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Is Space-Time Attention All You Need for Video Understanding?
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Zero-Shot Text-to-Image Generation
• Linformer: Self-Attention with Linear Complexity
• Rethinking the Inception Architecture for Computer Vision
• Generating Long Sequences with Sparse Transformers
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• A Short Note about Kinetics-600
• Stand-Alone Self-Attention in Vision Models
• Object-Centric Learning with Slot Attention
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• Toward Transformer-Based Object Detection
• Training Vision Transformers for Image Retrieval
• Rethinking Attention with Performers
• CPTR: Full Transformer Network for Image Captioning
• Understanding Back-Translation at Scale
• TransReID: Transformer-based Object Re-Identification
• End-to-End Video Instance Segmentation with Transformers
• How to Start Training: The Effect of Initialization and Architecture
• End-to-End Human Pose and Mesh Reconstruction with Transformers
• SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation
• Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation

Cited by in corpus (35)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• XCiT: Cross-Covariance Image Transformers
• ActionCLIP: A New Paradigm for Video Action Recognition
• ResT: An Efficient Transformer for Visual Recognition
• HRFormer: High-Resolution Transformer for Dense Prediction
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• CycleMLP: A MLP-like Architecture for Dense Prediction
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Benchmarking Detection Transfer Learning with Vision Transformers
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Video Swin Transformer
• Glance-and-Gaze Vision Transformer
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• Fully Transformer Networks for Semantic Image Segmentation
• Video Action Understanding
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• Visual Parser: Representing Part-whole Hierarchies with Transformers
• Revisiting 3D ResNets for Video Recognition
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• Relation Modeling in Spatio-Temporal Action Localization
• Shifted Chunk Transformer for Spatio-Temporal Representational Learning
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• VidTr: Video Transformer Without Convolutions
• StarVQA: Space-Time Attention for Video Quality Assessment
• Long-Short Temporal Contrastive Learning of Video Transformers
• Efficient Video Transformers with Spatial-Temporal Token Selection
• Class Semantics-based Attention for Action Detection
• Overview of Tencent Multi-modal Ads Video Understanding Challenge
• PyTorchVideo: A Deep Learning Library for Video Understanding
• "Knights": First Place Submission for VIPriors21 Action Recognition Challenge at ICCV 2021
• The Efficiency Misnomer