ABNIRML: Analyzing the Behavior of Neural IR Models

Abstract

Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc search. However, it is not yet well-understood why these methods are so effective, what makes some variants more effective than others, and what pitfalls they may have. We present a new comprehensive framework for Analyzing the Behavior of Neural IR ModeLs (ABNIRML), which includes new types of diagnostic probes that allow us to test several characteristics -- such as writing styles, factuality, sensitivity to paraphrasing and word order -- that are not addressed by previous techniques. To demonstrate the value of the framework, we conduct an extensive empirical study that yields insights into the factors that contribute to the neural model's gains, and identify potential unintended biases the models exhibit. Some of our results confirm conventional wisdom, like that recent neural ranking models rely less on exact term overlap with the query, and instead leverage richer linguistic information, evidenced by their higher sensitivity to word and sentence order. Other results are more surprising, such as that some models (e.g., T5 and ColBERT) are biased towards factually correct (rather than simply relevant) texts. Further, some characteristics vary even for the same base language model, and other characteristics can appear due to random variations during model training.

arXiv commentTACL version

References in corpus (18)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• Passage Re-ranking with BERT
• CEDR: Contextualized Embeddings for Document Ranking
• Document Expansion by Query Prediction
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Declarative Experimentation in Information Retrieval using PyTerrier
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Simplified Data Wrangling with ir_datasets
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Overview of the TREC 2019 deep learning track
• Expansion via Prediction of Importance with Contextualization
• BERT Rediscovers the Classical NLP Pipeline
• PARADE: Passage Representation Aggregation for Document Reranking
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
• Content-Based Weak Supervision for Ad-Hoc Re-Ranking
• Diagnosing BERT with Retrieval Heuristics
• Towards Axiomatic Explanations for Neural Ranking Models

Cited by in corpus (5)

• Pretrained Transformers for Text Ranking: BERT and Beyond
• The Role of Complex NLP in Transformers for Text Ranking?
• Neural Passage Quality Estimation for Static Pruning
• Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval
• On the Effects of Regional Spelling Conventions in Retrieval Models