Temporal Query Networks for Fine-grained Video Understanding

Abstract

Our objective in this work is fine-grained classification of actions in untrimmed videos, where the actions may be temporally extended or may span only a few frames of the video. We cast this into a query-response mechanism, where each query addresses a particular question, and has its own response label set. We make the following four contributions: (I) We propose a new model - a Temporal Query Network - which enables the query-response functionality, and a structural understanding of fine-grained actions. It attends to relevant segments for each query with a temporal attention mechanism, and can be trained using only the labels for each query. (ii) We propose a new way - stochastic feature bank update - to train a network on videos of various lengths with the dense sampling required to respond to fine-grained queries. (iii) We compare the TQN to other architectures and text supervision methods, and analyze their pros and cons. Finally, (iv) we evaluate the method extensively on the FineGym and Diving48 benchmarks for fine-grained action classification and surpass the state-of-the-art using only RGB features.

Accepted to CVPR 2021(Oral). Project page: http://www.robots.ox.ac.uk/~vgg/research/tqn/

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Contrastive Multiview Coding
• Learnable pooling with Context Gating for video classification
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Learning Video Representations using Contrastive Bidirectional Transformer
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Video Representation Learning with Visual Tempo Consistency
• Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
• Learning Discriminative Motion Features Through Detection
• Progressive Attention Memory Network for Movie Story Question Answering
• Learning to Localize and Align Fine-Grained Actions to Sparse Instructions