Use What You Have: Video Retrieval Using Representations From Collaborative Experts

Abstract

The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge. Human-generated queries for video datasets `in the wild' vary a lot in terms of degree of specificity, with some queries describing specific details such as the names of famous identities, content from speech, or text available on the screen. Our goal is to condense the multi-modal, extremely high dimensional information from videos into a single, compact video representation for the task of video retrieval using free-form text queries, where the degree of specificity is open-ended. For this we exploit existing knowledge in the form of pre-trained semantic embeddings which include 'general' features such as motion, appearance, and scene features from visual content. We also explore the use of more 'specific' cues from ASR and OCR which are intermittently available for videos and find that these signals remain challenging to use effectively for retrieval. We propose a collaborative experts model to aggregate information from these different pre-trained experts and assess our approach empirically on five retrieval benchmarks: MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet. Code and data can be found at www.robots.ox.ac.uk/~vgg/research/collaborative-experts/. This paper contains a correction to results reported in the previous version.

This update contains a correction to previously reported results

References in corpus (4)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language

Cited by in corpus (39)

• Dual Encoding for Video Retrieval by Text
• MDMMT: Multidomain Multimodal Transformer for Video Retrieval
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• A Comprehensive Study of Deep Video Action Recognition
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss
• Support-set bottlenecks for video-text representation learning
• Text-based Localization of Moments in a Video Corpus
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
• On the adequacy of untuned warmup for adaptive optimization
• The complementarity of a diverse range of deep learning features extracted from video content for video recommendation
• MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
• Learn to Understand Negation in Video Retrieval
• HANet: Hierarchical Alignment Networks for Video-Text Retrieval
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• Semantic Role Aware Correlation Transformer for Text to Video Retrieval
• Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• Data leakage in cross-modal retrieval training: A case study
• An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video-Text Retrieval
• Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
• Temporal Query Networks for Fine-grained Video Understanding
• TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding
• TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
• SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
• Rethinking movie genre classification with fine-grained semantic clustering
• Universal Weighting Metric Learning for Cross-Modal Matching
• Are you Struggling? Dataset and Baselines for Struggle Determination in Assembly Videos
• Rudder: A Cross Lingual Video and Text Retrieval Dataset
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Multi-modal Transformer for Video Retrieval
• Video Moment Retrieval via Natural Language Queries
• Video-aided Unsupervised Grammar Induction