Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention

Abstract

Automatically describing video, or video captioning, has been widely studied in the multimedia field. This paper proposes a new task of sensor-augmented egocentric-video captioning, a newly constructed dataset for it called MMAC Captions, and a method for the newly proposed task that effectively utilizes multi-modal data of video and motion sensors, or inertial measurement units (IMUs). While conventional video captioning tasks have difficulty in dealing with detailed descriptions of human activities due to the limited view of a fixed camera, egocentric vision has greater potential to be used for generating the finer-grained descriptions of human activities on the basis of a much closer view. In addition, we utilize wearable-sensor data as auxiliary information to mitigate the inherent problems in egocentric vision: motion blur, self-occlusion, and out-of-camera-range activities. We propose a method for effectively utilizing the sensor data in combination with the video data on the basis of an attention mechanism that dynamically determines the modality that requires more attention, taking the contextual information into account. We compared the proposed sensor-fusion method with strong baselines on the MMAC Captions dataset and found that using sensor data as supplementary information to the egocentric-video data was beneficial, and that our proposed method outperformed the strong baselines, demonstrating the effectiveness of the proposed method.

Accepted to ACM Multimedia (ACMMM) 2021

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• The Evolution of First Person Vision Methods: A Survey
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Phrase-based Image Captioning
• Attention-Based Multimodal Fusion for Video Description
• MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
• A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
• DeepDiary: Automatic Caption Generation for Lifelogging Image Streams
• SBAT: Video Captioning with Sparse Boundary-Aware Transformer
• Symbiotic Attention with Privileged Information for Egocentric Action Recognition