Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention

Abstract

Automatically describing video, or video captioning, has been widely studied in the multimedia field. This paper proposes a new task of sensor-augmented egocentric-video captioning, a newly constructed dataset for it called MMAC Captions, and a method for the newly proposed task that effectively utilizes multi-modal data of video and motion sensors, or inertial measurement units (IMUs). While conventional video captioning tasks have difficulty in dealing with detailed descriptions of human activities due to the limited view of a fixed camera, egocentric vision has greater potential to be used for generating the finer-grained descriptions of human activities on the basis of a much closer view. In addition, we utilize wearable-sensor data as auxiliary information to mitigate the inherent problems in egocentric vision: motion blur, self-occlusion, and out-of-camera-range activities. We propose a method for effectively utilizing the sensor data in combination with the video data on the basis of an attention mechanism that dynamically determines the modality that requires more attention, taking the contextual information into account. We compared the proposed sensor-fusion method with strong baselines on the MMAC Captions dataset and found that using sensor data as supplementary information to the egocentric-video data was beneficial, and that our proposed method outperformed the strong baselines, demonstrating the effectiveness of the proposed method.

arXiv commentAccepted to ACM Multimedia (ACMMM) 2021

References in corpus (26)

• Distilling the Knowledge in a Neural Network
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Categorical Reparameterization with Gumbel-Softmax
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• The Evolution of First Person Vision Methods: A Survey
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Hierarchical Boundary-Aware Neural Encoder for Video Captioning
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• EgoSampling: Fast-Forward and Stereo for Egocentric Videos
• Phrase-based Image Captioning
• Going Deeper into First-Person Activity Recognition
• Less Is More: Picking Informative Frames for Video Captioning
• Attention-Based Multimodal Fusion for Video Description
• Attributes' Importance for Zero-Shot Pose-Classification Based on Wearable Sensors
• MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
• Ego-Pose Estimation and Forecasting as Real-Time PD Control
• A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
• Reflective Decoding Network for Image Captioning
• DeepDiary: Automatic Caption Generation for Lifelogging Image Streams
• SBAT: Video Captioning with Sparse Boundary-Aware Transformer
• Symbiotic Attention with Privileged Information for Egocentric Action Recognition