Learning-to-Rank with BERT in TF-Ranking
arXiv:2004.08476
Abstract
This paper describes a machine learning algorithm for document (re)ranking, in which queries and documents are firstly encoded using BERT [1], and on top of that a learning-to-rank (LTR) model constructed with TF-Ranking (TFR) [2] is applied to further optimize the ranking performance. This approach is proved to be effective in a public MS MARCO benchmark [3]. Our first two submissions achieve the best performance for the passage re-ranking task [4], and the second best performance for the passage full-ranking task as of April 10, 2020 [5]. To leverage the lately development of pre-trained language models, we recently integrate RoBERTa [6] and ELECTRA [7]. Our latest submissions improve our previously state-of-the-art re-ranking performance by 4.3% [8], and achieve the third best performance for the full-ranking task [9] as of June 8, 2020. Both of them demonstrate the effectiveness of combining ranking losses with BERT representations for document ranking.
arXiv comment6 pages, 1 figure, 2 tables
References in corpus (2)
Cited by in corpus (14)
- The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
- RRF102: Meeting the TREC-COVID Challenge with a 100+ Runs Ensemble
- Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
- Optimizing Dense Retrieval Model Training with Hard Negatives
- Fairness Through Regularization for Learning to Rank
- Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
- Argument Undermining: Counter-Argument Generation by Attacking Weak Premises
- Explicit and Implicit Semantic Ranking Framework
- Ranking Creative Language Characteristics in Small Data Scenarios
- Knowledge-driven Answer Generation for Conversational Search
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
- Learning Representations for Axis-Aligned Decision Forests through Input Perturbation
- "It doesn't look good for a date": Transforming Critiques into Preferences for Conversational Recommendation Systems
- Red Dragon AI at TextGraphs 2021 Shared Task: Multi-Hop Inference Explanation Regeneration by Matching Expert Ratings