PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
arXiv:2312.02429 · doi:10.1145/3616855.3635791
Abstract
Embedding-based Retrieval Models (ERMs) have emerged as a promising framework for large-scale text retrieval problems due to powerful large language models. Nevertheless, fine-tuning ERMs to reach state-of-the-art results can be expensive due to the extreme scale of data as well as the complexity of multi-stages pipelines (e.g., pre-training, fine-tuning, distillation). In this work, we propose the PEFA framework, namely ParamEter-Free Adapters, for fast tuning of ERMs without any backward pass in the optimization. At index building stage, PEFA equips the ERM with a non-parametric k-nearest neighbor (kNN) component. At inference stage, PEFA performs a convex combination of two scoring functions, one from the ERM and the other from the kNN. Based on the neighborhood definition, PEFA framework induces two realizations, namely PEFA-XL (i.e., extra large) using double ANN indices and PEFA-XS (i.e., extra small) using a single ANN index. Empirically, PEFA achieves significant improvement on two retrieval applications. For document retrieval, regarding Recall@100 metric, PEFA improves not only pre-trained ERMs on Trivia-QA by an average of 13.2%, but also fine-tuned ERMs on NQ-320K by an average of 5.5%, respectively. For product search, PEFA improves the Recall@100 of the fine-tuned ERMs by an average of 5.3% and 14.5%, for PEFA-XS and PEFA-XL, respectively. Our code is available at https://github.com/amzn/pecos/tree/mainline/examples/pefa-wsdm24.
Accept by WSDM 2024
References in corpus (20)
- LoRA: Low-Rank Adaptation of Large Language Models
- Simplifying Graph Convolutional Networks
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
- Improving language models by retrieving from trillions of tokens
- Embedding-based Retrieval in Facebook Search
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Semantic Models for the First-stage Retrieval: A Comprehensive Review
- Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
- Transformer Memory as a Differentiable Search Index
- Autoregressive Search Engines: Generating Substrings as Document Identifiers
- A Neural Corpus Indexer for Document Retrieval
- Adversarial Retriever-Ranker for dense text retrieval
- PECOS: Prediction for Enormous and Correlated Output Spaces
- Large-Scale Training System for 100-Million Classification at Alibaba
- A Greedy Approach for Budgeted Maximum Inner Product Search
- From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective
- Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
- A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Search
- Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval
- DPTDR: Deep Prompt Tuning for Dense Passage Retrieval