In the Eye of the Beholder: Gaze and Actions in First Person Video
arXiv:2006.00626
Abstract
We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. To facilitate our research, we first introduce the EGTEA Gaze+ dataset. Our dataset comes with videos, gaze tracking data, hand masks and action annotations, thereby providing the most comprehensive benchmark for First Person Vision (FPV). Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV. Our method describes the participant's gaze as a probabilistic variable and models its distribution using stochastic units in a deep network. We further sample from these stochastic units, generating an attention map to guide the aggregation of visual features for action recognition. Our method is evaluated on our EGTEA Gaze+ dataset and achieves a performance level that exceeds the state-of-the-art by a significant margin. More importantly, we demonstrate that our model can be applied to larger scale FPV dataset---EPIC-Kitchens even without using gaze, offering new state-of-the-art results on FPV action recognition.
arXiv commentSubmitted to TPAMI
References in corpus (18)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Auto-Encoding Variational Bayes
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Categorical Reparameterization with Gumbel-Softmax
- Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- DropBlock: A regularization method for convolutional networks
- Action Recognition using Visual Attention
- The Evolution of First Person Vision Methods: A Survey
- Analysis of the hands in egocentric vision: A survey
- Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
- Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions
- Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition
- Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
- Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video
- Attention Distillation for Learning Video Representations