Support-set bottlenecks for video-text representation learning
arXiv:2010.02824
Abstract
The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semantically-related -- for example, visually similar videos or ones that share the same depicted action. In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample's caption must be reconstructed as a weighted combination of other support samples' visual representations. This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX and ActivityNet, and MSVD for video-to-text and text-to-video retrieval.
arXiv commentAccepted as spotlight paper at the International Conference on Learning Representations (ICLR) 2021
References in corpus (23)
- Efficient Estimation of Word Representations in Vector Space
- Sequence to Sequence Learning with Neural Networks
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- YFCC100M: The New Data in Multimedia Research
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Prototypical Contrastive Learning of Unsupervised Representations
- Large Scale Adversarial Representation Learning
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
- Use What You Have: Video Retrieval Using Representations From Collaborative Experts
- Learning Video Representations using Contrastive Bidirectional Transformer
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Self-Supervised MultiModal Versatile Networks
- UNITER: UNiversal Image-TExt Representation Learning
- Pre-training via Paraphrasing
- Labelling unlabelled videos from scratch with multi-modal self-supervision
- Object Relational Graph with Teacher-Recommended Learning for Video Captioning
- Jointly Localizing and Describing Events for Dense Video Captioning
- Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
- Omni-sourced Webly-supervised Learning for Video Recognition