Contrastive Fine-tuning Improves Robustness for Neural Rankers

Abstract

The performance of state-of-the-art neural rankers can deteriorate substantially when exposed to noisy inputs or applied to a new domain. In this paper, we present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. Specifically, a contrastive loss that compares data points in the representation space is combined with the standard ranking loss during fine-tuning. We use relevance labels to denote similar/dissimilar pairs, which allows the model to learn the underlying matching semantics across different query-document pairs and leads to improved robustness. In experiments with four passage ranking datasets, the proposed contrastive fine-tuning method obtains improvements on robustness to query reformulations, noise perturbations, and zero-shot transfer for both BERT and BART based rankers. Additionally, our experiments show that contrastive fine-tuning outperforms data augmentation for robustifying neural rankers.

References in corpus (13)

• Learning Transferable Visual Models From Natural Language Supervision
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Contrastive Representation Learning: A Framework and Review
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• CLEAR: Contrastive Learning for Sentence Representation
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Neural Passage Retrieval with Improved Negative Contrast
• PyTorch Metric Learning
• Passage Ranking with Weak Supervision
• Benchmarking Robustness of Machine Reading Comprehension Models
• BERTSel: Answer Selection with Pre-trained Models