Forecasting Action through Contact Representations from First Person Video

Abstract

Human actions involving hand manipulations are structured according to the making and breaking of hand-object contact, and human visual understanding of action is reliant on anticipation of contact as is demonstrated by pioneering work in cognitive science. Taking inspiration from this, we introduce representations and models centered on contact, which we then use in action prediction and anticipation. We annotate a subset of the EPIC Kitchens dataset to include time-to-contact between hands and objects, as well as segmentations of hands and objects. Using these annotations we train the Anticipation Module, a module producing Contact Anticipation Maps and Next Active Object Segmentations - novel low-level representations providing temporal and spatial characteristics of anticipated near future action. On top of the Anticipation Module we apply Egocentric Object Manipulation Graphs (Ego-OMG), a framework for action anticipation and prediction. Ego-OMG models longer term temporal semantic relations through the use of a graph modeling transitions between contact delineated action states. Use of the Anticipation Module within Ego-OMG produces state-of-the-art results, achieving 1st and 2nd place on the unseen and seen test sets, respectively, of the EPIC Kitchens Action Anticipation Challenge, and achieving state-of-the-art results on the tasks of action anticipation and action prediction over EPIC Kitchens. We perform ablation studies over characteristics of the Anticipation Module to evaluate their utility.

12 pages, 5 figures. in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Simple Online and Realtime Tracking
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
• Human Action Recognition and Prediction: A Survey
• Next-Active-Object prediction from Egocentric Videos
• Object Level Visual Reasoning in Videos
• Egocentric Object Manipulation Graphs
• Human Motion Anticipation with Symbolic Label

Cited by in corpus (7)

• Learning to Discriminate Information for Online Action Detection: Analysis and Application
• Semantically Guided Representation Learning For Action Anticipation
• Streaming egocentric action anticipation: An evaluation scheme and approach
• Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction
• Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectives
• Weakly-Supervised Dense Action Anticipation
• Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videos