Object Level Visual Reasoning in Videos

Abstract

Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object classes present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of understanding that pushes beyond this and call for models with capabilities for fine distinction and detailed comprehension of interactions between actors and objects in a scene. We propose a model capable of learning to reason about semantically meaningful spatiotemporal interactions in videos. The key to our approach is a choice of performing this reasoning at the object level through the integration of state of the art object detection networks. This allows the model to learn detailed spatial interactions that exist at a semantic, object-interaction relevant level. We evaluate our method on three standard datasets (Twenty-BN Something-Something, VLOG and EPIC Kitchens) and achieve state of the art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their interactions corresponding to different activity classes.

Accepted at ECCV 2018 - long version (16 pages + ref)

References in corpus (7)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• A simple neural network module for relational reasoning
• Action Recognition using Visual Attention
• FiLM: Visual Reasoning with a General Conditioning Layer
• Compositional Attention Networks for Machine Reasoning
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions

Cited by in corpus (28)

• Audiovisual SlowFast Networks for Video Recognition
• Forecasting Action through Contact Representations from First Person Video
• Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors
• CoPhy: Counterfactual Learning of Physical Dynamics
• Long-Term Feature Banks for Detailed Video Understanding
• What Makes Training Multi-Modal Classification Networks Hard?
• Grouped Spatial-Temporal Aggregation for Efficient Action Recognition
• Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries
• Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
• Action Genome: Actions as Composition of Spatio-temporal Scene Graphs
• Detecting Human-Object Interactions with Action Co-occurrence Priors
• Interactive Fusion of Multi-level Features for Compositional Activity Recognition
• Relational Action Forecasting
• Understanding Human Hands in Contact at Internet Scale
• Asynchronous Interaction Aggregation for Action Detection
• Domain-Specific Priors and Meta Learning for Few-Shot First-Person Action Recognition
• Analogical Reasoning for Visually Grounded Language Acquisition
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• ACP++: Action Co-occurrence Priors for Human-Object Interaction Detection
• A Structured Model For Action Detection
• SAFCAR: Structured Attention Fusion for Compositional Action Recognition
• Relation Network for Person Re-identification
• Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
• Symbiotic Attention with Privileged Information for Egocentric Action Recognition
• Recognizing Manipulation Actions from State-Transformations
• Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language Tasks
• A Self Validation Network for Object-Level Human Attention Estimation
• SemCo: Toward Semantic Coherent Visual Relationship Forecasting