Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos

Abstract

We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e.g., "dressing") to the action (e.g., "mix yogurt") that produced it. The key challenge is the inevitable visual-linguistic ambiguities arising from the changes in both visual appearance and referring expression of an entity in the video. This challenge is amplified by the fact that we aim to resolve references with no supervision. We address these challenges by learning a joint visual-linguistic model, where linguistic cues can help resolve visual ambiguities and vice versa. We verify our approach by learning our model unsupervisedly using more than two thousand unstructured cooking videos from YouTube, and show that our visual-linguistic model can substantially improve upon state-of-the-art linguistic only model on reference resolution in instructional videos.

CVPR 2017

References in corpus (7)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Visual Storytelling
• Modeling Context in Referring Expressions
• Modeling Relationships in Referential Expressions with Compositional Modular Networks
• Learning Visual Storylines with Skipping Recurrent Neural Networks
• Joint Discovery of Object States and Manipulation Actions

Cited by in corpus (3)

• Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
• Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
• How to Make a BLT Sandwich? Learning to Reason towards Understanding Web Instructional Videos