Identifying Visible Actions in Lifestyle Vlogs

Abstract

We consider the task of identifying human actions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech description of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages information derived from visual and linguistic clues to automatically infer which actions are visible in a video. We demonstrate that our multimodal algorithm outperforms algorithms based only on one modality at a time.

Accepted at ACL 2019

References in corpus (5)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding

Cited by in corpus (2)

• What is More Likely to Happen Next? Video-and-Language Future Event Prediction
• When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs