Multiscale Vision Transformers

Abstract

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10x more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast

Technical report

References in corpus (14)

• Improving neural networks by preventing co-adaptation of feature detectors
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Learning Transferable Visual Models From Natural Language Supervision
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• Zero-Shot Text-to-Image Generation
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• Stand-Alone Self-Attention in Vision Models
• Training Vision Transformers for Image Retrieval
• TransReID: Transformer-based Object Re-Identification
• SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation
• Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation