COBE: Contextualized Object Embeddings from Narrated Instructional Video
arXiv:2007.07306
Abstract
Many objects in the real world undergo dramatic variations in visual appearance. For example, a tomato may be red or green, sliced or chopped, fresh or fried, liquid or solid. Training a single detector to accurately recognize tomatoes in all these different states is challenging. On the other hand, contextual cues (e.g., the presence of a knife, a cutting board, a strainer or a pan) are often strongly indicative of how the object appears in the scene. Recognizing such contextual cues is useful not only to improve the accuracy of object detection or to determine the state of the object, but also to understand its functional properties and to infer ongoing or upcoming human-object interactions. A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms. Instead of relying on manually-labeled data for this task, we propose a new framework for learning Contextualized OBject Embeddings (COBE) from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. This enables the learning of an object representation where concepts relate according to a semantic language metric. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.
arXiv commentNeurIPS 2020
References in corpus (12)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distributed Representations of Words and Phrases and their Compositionality
- Representation Learning with Contrastive Predictive Coding
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- R-FCN: Object Detection via Region-based Fully Convolutional Networks
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Billion-scale semi-supervised learning for image classification
- Self-training with Noisy Student improves ImageNet classification
- Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
- Jointly Modeling Embedding and Translation to Bridge Video and Language