Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

Abstract

Supervised machine learning models and their evaluation strongly depends on the quality of the underlying dataset. When we search for a relevant piece of information it may appear anywhere in a given passage. However, we observe a bias in the position of the correct answer in the text in two popular Question Answering datasets used for passage re-ranking. The excessive favoring of earlier positions inside passages is an unwanted artefact. This leads to three common Transformer-based re-ranking models to ignore relevant parts in unseen passages. More concerningly, as the evaluation set is taken from the same biased distribution, the models overfitting to that bias overestimate their true effectiveness. In this work we analyze position bias on datasets, the contextualized representations, and their effect on retrieval results. We propose a debiasing method for retrieval datasets. Our results show that a model trained on a position-biased dataset exhibits a significant decrease in re-ranking effectiveness when evaluated on a debiased dataset. We demonstrate that by mitigating the position bias, Transformer-based re-ranking models are equally effective on a biased and debiased dataset, as well as more effective in a transfer-learning setting between two differently biased datasets.

arXiv commentAccepted at ECIR 2021 (Full paper track)

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Towards A Rigorous Science of Interpretable Machine Learning
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Passage Re-ranking with BERT
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Sparse, Dense, and Attentional Representations for Text Retrieval
• Overview of the TREC 2021 deep learning track
• Evaluation Metrics for Measuring Bias in Search Engine Results
• Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
• Towards Robust and Privacy-preserving Text Representations
• Compositional Questions Do Not Necessitate Multi-hop Reasoning
• Let's measure run time! Extending the IR replicability infrastructure to include performance aspects
• Self-Attention with Structural Position Representations
• Assessing the Ability of Self-Attention Networks to Learn Word Order