Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models

Abstract

Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed skepticism that neural ranking models were actually improving ad hoc retrieval effectiveness in limited data scenarios. He provided anecdotal evidence that authors of neural IR papers demonstrate "wins" by comparing against weak baselines. This paper provides a rigorous evaluation of those claims in two ways: First, we conducted a meta-analysis of papers that have reported experimental results on the TREC Robust04 test collection. We do not find evidence of an upward trend in effectiveness over time. In fact, the best reported results are from a decade ago and no recent neural approach comes close. Second, we applied five recent neural models to rerank the strong baselines that Lin used to make his arguments. A significant improvement was observed for one of the models, demonstrating additivity in gains. While there appears to be merit to neural IR approaches, at least some of the gains reported in the literature appear illusory.

arXiv commentPublished in the Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019)

References in corpus (5)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• End-to-End Open-Domain Question Answering with BERTserini
• CEDR: Contextualized Embeddings for Document Ranking
• Neural Vector Spaces for Unsupervised Information Retrieval

Cited by in corpus (19)

• A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
• Empirical Analysis of Session-Based Recommendation Algorithms
• Information Retrieval: Recent Advances and Beyond
• Keyphrase Generation for Scientific Document Retrieval
• ir_metadata: An Extensible Metadata Schema for IR Experiments
• Towards Axiomatic Explanations for Neural Ranking Models
• Selective Weak Supervision for Neural Information Retrieval
• All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes
• Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems
• Training Curricula for Open Domain Answer Re-Ranking
• "Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time
• CsFEVER and CTKFacts: Acquiring Czech data for fact verification
• Long Document Re-ranking with Modular Re-ranker
• Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Language
• Overview of LiLAS 2020 -- Living Labs for Academic Search
• An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering
• Living Lab Evaluation for Life and Social Sciences Search Platforms -- LiLAS at CLEF 2021
• ReFormeR: Learning and Applying Explicit Query Reformulation Patterns
• GRIT: Graph-based Recall Improvement for Task-oriented E-commerce Queries