CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations

Abstract

Contrastive learning allows us to flexibly define powerful losses by contrasting positive pairs from sets of negative samples. Recently, the principle has also been used to learn cross-modal embeddings for video and text, yet without exploiting its full potential. In particular, previous losses do not take the intra-modality similarities into account, which leads to inefficient embeddings, as the same content is mapped to multiple points in the embedding space. With CrossCLR, we present a contrastive loss that fixes this issue. Moreover, we define sets of highly related samples in terms of their input embeddings and exclude them from the negative samples to avoid issues with false negatives. We show that these principles consistently improve the quality of the learned embeddings. The joint embeddings learned with CrossCLR extend the state of the art in video-text retrieval on Youcook2 and LSMDC datasets and in video captioning on Youcook2 dataset by a large margin. We also demonstrate the generality of the concept by learning improved joint embeddings for other pairs of modalities.

arXiv commentICCV 2021, 14 pages, 13 figures

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Self-supervised Co-training for Video Representation Learning
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Spatiotemporal Contrastive Video Representation Learning
• Instance Cross Entropy for Deep Metric Learning