Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos

Abstract

We introduce the task of spatially localizing narrated interactions in videos. Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations. To achieve this goal, we propose a multilayer cross-modal attention network that enables effective optimization of a contrastive loss during training. We introduce a divided strategy that alternates between computing inter- and intra-modal attention across the visual and natural language modalities, which allows effective training via directly contrasting the two modalities' representations. We demonstrate the effectiveness of our approach by self-training on the HowTo100M instructional video dataset and evaluating on a newly collected dataset of localized described interactions in the YouCook2 dataset. We show that our approach outperforms alternative baselines, including shallow co-attention and full cross-modal attention. We also apply our approach to grounding phrases in images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective and, when combined with a word-to-region loss, achieves state of the art on recall-at-one and pointing hand accuracies.

arXiv commentAccepted at NeurIPS 2021 (Spotlight)

References in corpus (13)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Transferable Visual Models From Natural Language Supervision
• Improved Baselines with Momentum Contrastive Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Learning Video Representations using Contrastive Bidirectional Transformer
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Space-Time Correspondence as a Contrastive Random Walk
• Self-supervised Video Representation Learning by Pace Prediction
• Distilling Localization for Self-Supervised Representation Learning
• Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
• COBE: Contextualized Object Embeddings from Narrated Instructional Video