Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

Abstract

A BERT-based Neural Ranking Model (NRM) can be either a crossencoder or a bi-encoder. Between the two, bi-encoder is highly efficient because all the documents can be pre-processed before the actual query time. In this work, we show two approaches for improving the performance of BERT-based bi-encoders. The first approach is to replace the full fine-tuning step with a lightweight fine-tuning. We examine lightweight fine-tuning methods that are adapter-based, prompt-based, and hybrid of the two. The second approach is to develop semi-Siamese models where queries and documents are handled with a limited amount of difference. The limited difference is realized by learning two lightweight fine-tuning modules, where the main language model of BERT is kept common for both query and document. We provide extensive experiment results for monoBERT, TwinBERT, and ColBERT where three performance metrics are evaluated over Robust04, ClueWeb09b, and MS-MARCO datasets. The results confirm that both lightweight fine-tuning and semi-Siamese are considerably helpful for improving BERT-based bi-encoders. In fact, lightweight fine-tuning is helpful for crossencoder, too

10 pages, 4 figures

References in corpus (11)

• Language Models are Few-Shot Learners
• LoRA: Low-Rank Adaptation of Large Language Models
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Revisiting Few-sample BERT Fine-tuning
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• How fine can fine-tuning be? Learning efficient language models
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

Cited by in corpus (2)

• Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval
• PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models