Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval
arXiv:2111.09276
Abstract
Schemata are structured representations of complex tasks that can aid artificial intelligence by allowing models to break down complex tasks into intermediate steps. We propose a novel system that induces schemata from web videos and generalizes them to capture unseen tasks with the goal of improving video retrieval performance. Our system proceeds in three major phases: (1) Given a task with related videos, we construct an initial schema for a task using a joint video-text model to match video segments with text representing steps from wikiHow; (2) We generalize schemata to unseen tasks by leveraging language models to edit the text within existing schemata. Through generalization, we can allow our schemata to cover a more extensive range of tasks with a small amount of learning data; (3) We conduct zero-shot instructional video retrieval with the unseen task names as the queries. Our schema-guided approach outperforms existing methods for video retrieval, and we demonstrate that the schemata induced by our system are better than those generated by other models.
References in corpus (11)
- Learning Transferable Visual Models From Natural Language Supervision
- Language Models are Few-Shot Learners
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Towards Automatic Learning of Procedures from Web Instructional Videos
- MDMMT: Multidomain Multimodal Transformer for Video Retrieval
- Compositional Attention Networks for Machine Reasoning
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- MERLOT: Multimodal Neural Script Knowledge Models
- Learning by Abstraction: The Neural State Machine
- ReFormer: The Relational Transformer for Image Captioning
- A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos