Deeper Text Understanding for IR with Contextual Neural Language Modeling
arXiv:1905.09217 · doi:10.1145/3331184.3331303
Abstract
Neural networks provide new possibilities to automatically learn complex language patterns and query-document relations. Neural IR models have achieved promising results in learning query-document relevance patterns, but few explorations have been done on understanding the text content of a query or a document. This paper studies leveraging a recently-proposed contextual neural language model, BERT, to provide deeper text understanding for IR. Experimental results demonstrate that the contextual text representations from BERT are more effective than traditional word embeddings. Compared to bag-of-words retrieval models, the contextual language model can better leverage language structures, bringing large improvements on queries written in natural languages. Combining the text understanding ability with search knowledge leads to an enhanced pre-trained BERT model that can benefit related search tasks where training data are limited.
In proceedings of SIGIR 2019
References in corpus (4)
Cited by in corpus (93)
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Information Retrieval: Recent Advances and Beyond
- At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence
- Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
- Pretrained Transformers for Text Ranking: BERT and Beyond
- Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
- Efficient Document Re-Ranking for Transformers by Precomputing Term Representations
- Complementing Lexical Retrieval with Semantic Residual Embedding
- Retrieval-Enhanced Machine Learning
- Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
- A Transformer-based Embedding Model for Personalized Product Search
- Table Search Using a Deep Contextualized Language Model
- PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval
- The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
- Neural ranking models for document retrieval
- Adversarial Retriever-Ranker for dense text retrieval
- StruBERT: Structure-aware BERT for Table Search and Matching
- C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval
- ABNIRML: Analyzing the Behavior of Neural IR Models
- Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models
- Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking
- Diagnosing BERT with Retrieval Heuristics
- MGeo: Multi-Modal Geographic Pre-Training Method
- Helping results assessment by adding explainable elements to the deep relevance matching model
- Towards Axiomatic Explanations for Neural Ranking Models
- Cross-lingual Information Retrieval with BERT
- Learning a Fine-Grained Review-based Transformer Model for Personalized Product Search
- Selective Weak Supervision for Neural Information Retrieval
- Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning
- A Multi-task Learning Framework for Product Ranking with BERT
- Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellings
- Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models
- Conformer-Kernel with Query Term Independence for Document Retrieval
- B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval
- Zero-shot Query Contextualization for Conversational Search
- A Linguistic Study on Relevance Modeling in Information Retrieval
- The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
- Graph-based Hierarchical Relevance Matching Signals for Ad-hoc Retrieval
- Efficient Neural Ranking using Forward Indexes
- P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
- Enhancing Relevance of Embedding-based Retrieval at Walmart
- SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
- Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
- Deep Reinforced Query Reformulation for Information Retrieval
- Composite Re-Ranking for Efficient Document Search with BERT
- "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"
- Certified Robustness to Word Substitution Ranking Attack for Neural Ranking Models
- Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
- Query Focused Multi-Document Summarization with Distant Supervision
- COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
- Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
- Relevance Filtering for Embedding-based Retrieval
- Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
- Neural Passage Quality Estimation for Static Pruning
- Semantic Labeling Using a Deep Contextualized Language Model
- Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
- Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset
- Distillation for Multilingual Information Retrieval
- Long Document Re-ranking with Modular Re-ranker
- Simplified TinyBERT: Knowledge Distillation for Document Retrieval
- Investigating the Effects of Sparse Attention on Cross-Encoders
- Transformer Based Language Models for Similar Text Retrieval and Ranking
- Query expansion with artificially generated texts
- On Single and Multiple Representations in Dense Passage Retrieval
- BERT-QE: Contextualized Query Expansion for Document Re-ranking
- Searching Scientific Literature for Answers on COVID-19 Questions
- Meta Learning to Rank for Sparsely Supervised Queries
- Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls
- PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval
- OpenMatch: An Open Source Library for Neu-IR Research
- IntenT5: Search Result Diversification using Causal Language Models
- Cross-Lingual Training with Dense Retrieval for Document Retrieval
- Long Document Ranking with Query-Directed Sparse Transformer
- Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm
- CWRCzech: 100M Query-Document Czech Click Dataset and Its Application to Web Relevance Ranking
- Query-driven Segment Selection for Ranking Long Documents
- Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis
- DeText: A Deep Text Ranking Framework with BERT
- An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
- Dealing with Typos for BERT-based Passage Retrieval and Ranking
- Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
- Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
- ConQX: Semantic Expansion of Spoken Queries for Intent Detection based on Conditioned Text Generation
- More Robust Dense Retrieval with Contrastive Dual Learning
- Deep Natural Language Processing for LinkedIn Search
- Cross-language Information Retrieval
- Searching Personal Collections
- Modularized Transfomer-based Ranking Framework
- Knowledge-Enhanced Evidence Retrieval for Counterargument Generation
- On the Reproducibility of Learned Sparse Retrieval Adaptations for Long Documents