Is Space-Time Attention All You Need for Video Understanding?

Abstract

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

Accepted to ICML 2021

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• A Short Note about Kinetics-600
• Image Transformer
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation

Cited by in corpus (112)

• Transformers in Vision: A Survey
• Human Action Recognition from Various Data Modalities: A Review
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Escaping the Big Data Paradigm with Compact Transformers
• End-to-end Temporal Action Detection with Transformer
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• XCiT: Cross-Covariance Image Transformers
• Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
• ActionCLIP: A New Paradigm for Video Action Recognition
• Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
• Video Transformers: A Survey
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
• HRFormer: High-Resolution Transformer for Dense Prediction
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• CycleMLP: A MLP-like Architecture for Dense Prediction
• RegionViT: Regional-to-Local Attention for Vision Transformers
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• ViTGAN: Training GANs with Vision Transformers
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• LoViT: Long Video Transformer for Surgical Phase Recognition
• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment
• ASFormer: Transformer for Action Segmentation
• Video Instance Segmentation using Inter-Frame Communication Transformers
• Space-time Mixing Attention for Video Transformer
• Multiscale Vision Transformers
• Video Swin Transformer
• Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training
• CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
• Vision Transformers with Patch Diversification
• Human in Events: A Large-Scale Benchmark for Human-centric Video Analysis in Complex Events
• Multimodal Fish Feeding Intensity Assessment in Aquaculture
• Sporthesia: Augmenting Sports Videos Using Natural Language
• Video Action Understanding
• SVFAP: Self-supervised Video Facial Affect Perceiver
• P2AT: Pyramid Pooling Axial Transformer for Real-time Semantic Segmentation
• Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention
• Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach
• ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
• Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams
• Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling
• Location-aware Adaptive Normalization: A Deep Learning Approach For Wildfire Danger Forecasting
• Saliency-Aware Spatio-Temporal Artifact Detection for Compressed Video Quality Assessment
• Actor-agnostic Multi-label Action Recognition with Multi-modal Query
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• Temporal Action Proposal Generation with Transformers
• Revisiting 3D ResNets for Video Recognition
• Deep Optics for Video Snapshot Compressive Imaging
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• TTPOINT: A Tensorized Point Cloud Network for Lightweight Action Recognition with Event Cameras
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering
• Learning Motion and Temporal Cues for Unsupervised Video Object Segmentation
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for Dynamic Facial Expression Recognition
• Long Short-Term Transformer for Online Action Detection
• iRAG: Advancing RAG for Videos with an Incremental Approach
• FE-Adapter: Adapting Image-based Emotion Classifiers to Videos
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Learning from Temporal Gradient for Semi-supervised Action Recognition
• FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis
• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Do Language Models Understand Time?
• Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition
• Video-Text Pre-training with Learned Regions
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• Shifted Chunk Transformer for Spatio-Temporal Representational Learning
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• Video Is Graph: Structured Graph Module for Video Action Recognition
• Styleformer: Transformer based Generative Adversarial Networks with Style Vector
• Temporal Shuffling for Defending Deep Action Recognition Models against Adversarial Attacks
• Self-supervised Video-centralised Transformer for Video Face Clustering
• Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• Self-supervised Video Object Segmentation by Motion Grouping
• GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer
• Time-Space Transformers for Video Panoptic Segmentation
• Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network
• From Detection to Action Recognition: An Edge-Based Pipeline for Robot Human Perception
• Spatiotemporal Object Detection for Improved Aerial Vehicle Detection in Traffic Monitoring
• VidTr: Video Transformer Without Convolutions
• SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding
• EAGLE: Egocentric AGgregated Language-video Engine
• Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention
• MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings
• Transformation Invariant Cancerous Tissue Classification Using Spatially Transformed DenseNet
• Sub-word Level Lip Reading With Visual Attention
• Tab-Attention: Self-Attention-based Stacked Generalization for Imbalanced Credit Default Prediction
• Exploring Stronger Feature for Temporal Action Localization
• Tracking Without Re-recognition in Humans and Machines
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• Class Semantics-based Attention for Action Detection
• Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• Efficient Video Transformers with Spatial-Temporal Token Selection
• Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
• A Stronger Baseline for Ego-Centric Action Detection
• VideoLightFormer: Lightweight Action Recognition using Transformers
• StarVQA: Space-Time Attention for Video Quality Assessment
• Long-Short Temporal Contrastive Learning of Video Transformers
• Round Outcome Prediction in VALORANT Using Tactical Features from Video Analysis
• "Knights": First Place Submission for VIPriors21 Action Recognition Challenge at ICCV 2021
• Enhancing Transformer for Video Understanding Using Gated Multi-Level Attention and Temporal Adversarial Training
• Human Activity Recognition in an Open World
• Deep set conditioned latent representations for action recognition
• SATVSR: Scenario Adaptive Transformer for Cross Scenarios Video Super-Resolution
• ProSTformer: Pre-trained Progressive Space-Time Self-attention Model for Traffic Flow Forecasting
• Technical Report: Disentangled Action Parsing Networks for Accurate Part-level Action Parsing
• Delta Sampling R-BERT for limited data and low-light action recognition
• Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories
• Ripple Attention for Visual Perception with Sub-quadratic Complexity
• Video, How Do Your Tokens Merge?