A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
arXiv:2005.00706
Abstract
Watching instructional videos are often used to learn about procedures. Video captioning is one way of automatically collecting such knowledge. However, it provides only an indirect, overall evaluation of multimodal models with no finer-grained quantitative measure of what they have learned. We propose instead, a benchmark of structured procedural knowledge extracted from cooking videos. This work is complementary to existing tasks, but requires models to produce interpretable structured knowledge in the form of verb-argument tuples. Our manually annotated open-vocabulary resource includes 356 instructional cooking videos and 15,523 video clip/sentence-level annotations. Our analysis shows that the proposed task is challenging and standard modeling approaches like unsupervised segmentation, semantic role labeling, and visual action detection perform poorly when forced to predict every action of a procedure in a structured form.
Accepted by NLP Beyond Text - First International Workshop on Natural Language Processing Beyond Text @ EMNLP 2020
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Learning Video Representations using Contrastive Bidirectional Transformer
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Simple BERT Models for Relation Extraction and Semantic Role Labeling
- How2: A Large-scale Dataset for Multimodal Language Understanding
- VideoBERT: A Joint Model for Video and Language Representation Learning
- Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
- HAKE: Human Activity Knowledge Engine
- Simpler Context-Dependent Logical Forms via Model Projections
- Procedure Planning in Instructional Videos
- Learning to Segment Actions from Observation and Narration