Space-time Mixing Attention for Video Transformer

Abstract

This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer's depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code will be made available.

arXiv commentUpdated results on SSv2

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• The Kinetics Human Action Video Dataset
• Self-Attention Generative Adversarial Networks
• Longformer: The Long-Document Transformer
• Is Space-Time Attention All You Need for Video Understanding?
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• TAM: Temporal Adaptive Module for Video Recognition
• $A^2$-Nets: Double Attention Networks

Cited by in corpus (6)

• Human Action Recognition from Various Data Modalities: A Review
• Video Transformers: A Survey
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• FE-Adapter: Adapting Image-based Emotion Classifiers to Videos
• StarVQA: Space-Time Attention for Video Quality Assessment
• SAIC_Cambridge-HuPBA-FBK Submission to the EPIC-Kitchens-100 Action Recognition Challenge 2021