VideoGraph: Recognizing Minutes-Long Human Activities in Videos

Abstract

Many human activities take minutes to unfold. To represent them, related works opt for statistical pooling, which neglects the temporal structure. Others opt for convolutional methods, as CNN and Non-Local. While successful in learning temporal concepts, they are short of modeling minutes-long temporal dependencies. We propose VideoGraph, a method to achieve the best of two worlds: represent minutes-long human activities and learn their underlying temporal structure. VideoGraph learns a graph-based representation for human activities. The graph, its nodes and edges are learned entirely from video datasets, making VideoGraph applicable to problems without node-level annotation. The result is improvements over related works on benchmarks: Epic-Kitchen and Breakfast. Besides, we demonstrate that VideoGraph is able to learn the temporal structure of human activities in minutes-long videos.

References in corpus (14)

• Scikit-learn: Machine Learning in Python
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• A Structured Self-attentive Sentence Embedding
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Rank Pooling for Action Recognition
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Learnable pooling with Context Gating for video classification
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Temporal Relational Reasoning in Videos
• Concurrent Activity Recognition with Multimodal CNN-LSTM Structure
• How to Make a BLT Sandwich? Learning to Reason towards Understanding Web Instructional Videos
• Action Recognition with Spatio-Temporal Visual Attention on Skeleton Image Sequences

Cited by in corpus (10)

• Activity Graph Transformer for Temporal Action Localization
• Coarse to Fine Multi-Resolution Temporal Convolutional Network
• Domain-Specific Priors and Meta Learning for Few-Shot First-Person Action Recognition
• PIC: Permutation Invariant Convolution for Recognizing Long-range Activities
• Better Set Representations For Relational Reasoning
• Temporal Relational Modeling with Self-Supervision for Action Segmentation
• TimeGate: Conditional Gating of Segments in Long-range Activities
• NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
• Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition
• PGT: A Progressive Method for Training Models on Long Videos