Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

Abstract

In Actor and Observer we introduced a dataset linking the first and third-person video understanding domains, the Charades-Ego Dataset. In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68,536 activity instances in 68.8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available. Charades-Ego furthermore shares activity classes, scripts, and methodology with the Charades dataset, that consist of additional 82.3 hours of third-person video with 66,500 activity instances. Charades-Ego has temporal annotations and textual descriptions, making it suitable for egocentric video classification, localization, captioning, and new tasks utilizing the cross-modal nature of the data.

References in corpus (2)

• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Cited by in corpus (15)

• ROAD: The ROad event Awareness Dataset for Autonomous Driving
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
• Video Action Understanding
• AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
• Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
• In the Eye of the Beholder: Gaze and Actions in First Person Video
• Source-free Video Domain Adaptation by Learning from Noisy Labels
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• Spatial-Temporal Alignment Network for Action Recognition and Detection
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video
• When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs
• Recent Progress in Appearance-based Action Recognition
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
• AGENT: A Benchmark for Core Psychological Reasoning
• Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset