TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment

Abstract

Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning. This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques. The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words. This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words. Second, a cascade sampling method is applied to generate a small set of hard negative examples for efficient loss estimation for multi-modal fusion layers. To validate the effectiveness of TACo, in our experiments we finetune pretrained models for a set of downstream tasks including text-video retrieval (YouCook2, MSR-VTT and ActivityNet), video action step localization (CrossTask), video action segmentation (COIN). The results show that our models attain consistent improvements across different experimental settings over previous methods, setting new state-of-the-art on three public text-video retrieval benchmarks of YouCook2, MSR-VTT and ActivityNet.

arXiv commentAccepted by ICCV 2021

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• A Simple Framework for Contrastive Learning of Visual Representations
• Deep Residual Learning for Image Recognition
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• The Kinetics Human Action Video Dataset
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Learning Video Representations using Contrastive Bidirectional Transformer
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Image similarity using Deep CNN and Curriculum Learning
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• TVQA: Localized, Compositional Video Question Answering