MDMMT: Multidomain Multimodal Transformer for Video Retrieval
arXiv:2103.10699 · doi:10.1109/CVPRW53098.2021.00374
Abstract
We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.
References in corpus (15)
- Deep Residual Learning for Image Recognition
- The Kinetics Human Action Video Dataset
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- Use What You Have: Video Retrieval Using Representations From Collaborative Experts
- Predicting Visual Features from Text for Image and Video Caption Retrieval
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
- VideoBERT: A Joint Model for Video and Language Representation Learning
- Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
- Learning Language-Visual Embedding for Movie Understanding with Natural-Language
- A Dataset for Movie Description
- TRECVID 2020: A comprehensive campaign for evaluating video retrieval tasks across multiple application domains
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval
Cited by in corpus (10)
- CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- CycleMLP: A MLP-like Architecture for Dense Prediction
- Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss
- Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach
- Learning to Retrieve Videos by Asking Questions
- Learning to Locate Visual Answer in Video Corpus Using Question
- Learn to Understand Negation in Video Retrieval
- Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval
- Connecting Language and Vision for Natural Language-Based Vehicle Retrieval