LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
arXiv:2103.08784
Abstract
Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computation cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by pre-training on three novel learning objectives, extracting feature indexes offline, and employing instant dot-product matching with further re-ranking, which significantly speeds up retrieval process. In fact, LightningDOT achieves new state of the art across multiple ITR benchmarks such as Flickr30k, COCO and Multi30K, outperforming existing pre-trained models that consume 1000x magnitude of computational hours. Code and pre-training checkpoints are available at https://github.com/intersun/LightningDOT.
NAACL 2021
References in corpus (13)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Decoupled Weight Decay Regularization
- Learning Transferable Visual Models From Natural Language Supervision
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Efficient Natural Language Response Suggestion for Smart Reply
- Generation-Augmented Retrieval for Open-domain Question Answering
- Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators