Selective Weak Supervision for Neural Information Retrieval

Abstract

This paper democratizes neural information retrieval to scenarios where large scale relevance training signals are not available. We revisit the classic IR intuition that anchor-document relations approximate query-document relevance and propose a reinforcement weak supervision selection method, ReInfoSelect, which learns to select anchor-document pairs that best weakly supervise the neural ranker (action), using the ranking performance on a handful of relevance labels as the reward. Iteratively, for a batch of anchor-document pairs, ReInfoSelect back propagates the gradients through the neural ranker, gathers its NDCG reward, and optimizes the data selection network using policy gradients, until the neural ranker's performance peaks on target relevance metrics (convergence). In our experiments on three TREC benchmarks, neural rankers trained by ReInfoSelect, with only publicly available anchor data, significantly outperform feature-based learning to rank methods and match the effectiveness of neural rankers trained with private commercial search logs. Our analyses show that ReInfoSelect effectively selects weak supervision signals based on the stage of the neural ranker training, and intuitively picks anchor-document pairs similar to query-document pairs.

Accepted by WWW 2020

References in corpus (12)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• CEDR: Contextualized Embeddings for Document Ranking
• DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval
• Word-Entity Duet Representations for Document Ranking
• Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models
• Generic Intent Representation in Web Search
• Content-Based Weak Supervision for Ad-Hoc Re-Ranking
• Learning to Selectively Transfer: Reinforced Transfer Learning for Deep Text Matching
• Towards Better Text Understanding and Retrieval through Kernel Entity Salience Modeling
• ReQA: An Evaluation for End-to-End Answer Retrieval Models

Cited by in corpus (4)

• P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
• Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
• Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
• Unsupervised Dense Retrieval Training with Web Anchors