A Multi-level Alignment Training Scheme for Video-and-Language Grounding
arXiv:2204.10938 · doi:10.1109/ICDMW58026.2022.00124
Abstract
To solve video-and-language grounding tasks, the key is for the network to understand the connection between the two modalities. For a pair of video and language description, their semantic relation is reflected by their encodings' similarity. A good multi-modality encoder should be able to well capture both inputs' semantics and encode them in the shared feature space where embedding distance gets properly translated into their semantic similarity. In this work, we focused on this semantic connection between video and language, and developed a multi-level alignment training scheme to directly shape the encoding process. Global and segment levels of video-language alignment pairs were designed, based on the information similarity ranging from high-level context to fine-grained semantics. The contrastive loss was used to contrast the encodings' similarities between the positive and negative alignment pairs, and to ensure the network is trained in such a way that similar information is encoded closely in the shared feature space while information of different semantics is kept apart. Our multi-level alignment training can be applied to various video-and-language grounding tasks. Together with the task-specific training loss, our framework achieved comparable performance to previous state-of-the-arts on multiple video QA and retrieval datasets.
arXiv commentAccepted at ICDM 2022 FOMO-VL workshop
References in corpus (11)
- Decoupled Weight Decay Regularization
- Representation Learning with Contrastive Predictive Coding
- The Kinetics Human Action Video Dataset
- Improved Baselines with Momentum Contrastive Learning
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
- LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering