Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers

Abstract

Our objective is language-based search of large-scale image and video datasets. For this task, the approach that consists of independently mapping text and vision to a joint embedding space, a.k.a. dual encoders, is attractive as retrieval scales and is efficient for billions of images using approximate nearest neighbour search. An alternative approach of using vision-text transformers with cross-attention gives considerable improvements in accuracy over the joint embeddings, but is often inapplicable in practice for large-scale retrieval given the cost of the cross-attention mechanisms required for each sample at test time. This work combines the best of both worlds. We make the following three contributions. First, we equip transformer-based models with a new fine-grained cross-attention architecture, providing significant improvements in retrieval accuracy whilst preserving scalability. Second, we introduce a generic approach for combining a Fast dual encoder model with our Slow but accurate transformer-based model via distillation and re-ranking. Finally, we validate our approach on the Flickr30K image dataset where we show an increase in inference speed by several orders of magnitude while having results competitive to the state of the art. We also extend our method to the video domain, improving the state of the art on the VATEX dataset.

Accepted to CVPR 2021

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Exploring the Limits of Language Modeling
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• A Short Note about Kinetics-600
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Self-Supervised MultiModal Versatile Networks
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Support-set bottlenecks for video-text representation learning

Cited by in corpus (12)

• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Video Transformers: A Survey
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Local-Global Context Aware Transformer for Language-Guided Video Segmentation
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• X-volution: On the unification of convolution and self-attention
• Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment
• Image Retrieval from Contextual Descriptions
• Cascaded Fast and Slow Models for Efficient Semantic Code Search
• Image-text matching for large-scale book collections
• SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval