Training Vision Transformers for Image Retrieval

Abstract

Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt vision transformers for generating image descriptors and train the resulting model with a metric learning objective, which combines a contrastive loss with a differential entropy regularizer. Our results show consistent and significant improvements of transformers over convolution-based approaches. In particular, our method outperforms the state of the art on several public benchmarks for category-level retrieval, namely Stanford Online Product, In-Shop and CUB-200. Furthermore, our experiments on ROxford and RParis also show that, in comparable settings, transformers are competitive for particular object retrieval, especially in the regime of short vector representations and low-resolution images.

References in corpus (4)

• Aggregating Deep Convolutional Features for Image Retrieval
• Stand-Alone Self-Attention in Vision Models
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Classification is a Strong Baseline for Deep Metric Learning

Cited by in corpus (13)

• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• XCiT: Cross-Covariance Image Transformers
• Localizing Objects with Self-Supervised Transformers and no Labels
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Feature Fusion Vision Transformer for Fine-Grained Visual Categorization
• CMT: Convolutional Neural Networks Meet Vision Transformers
• Multiscale Vision Transformers
• Transformer-Based Source-Free Domain Adaptation
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• Robust and Decomposable Average Precision for Image Retrieval
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling