Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Abstract

Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source code are available at http://coin-dataset.github.io.

arXiv commentAccepted by T-PAMI, journal version of COIN dataset arXiv:1903.02874

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• How transferable are features in deep neural networks?
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• Towards Automatic Learning of Procedures from Web Instructional Videos
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• Diversity-aware Multi-Video Summarization
• Procedure Planning in Instructional Videos

Cited by in corpus (6)

• Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
• Uncertainty-aware Score Distribution Learning for Action Quality Assessment
• Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method
• VILT: Video Instructions Linking for Complex Tasks
• Background-Click Supervision for Temporal Action Localization