Pre-trained Language Model for Web-scale Retrieval in Baidu Search
arXiv:2106.03373
Abstract
Retrieval is a crucial stage in web search that identifies a small set of query-relevant candidates from a billion-scale corpus. Discovering more semantically-related candidates in the retrieval stage is very promising to expose more high-quality results to the end users. However, it still remains non-trivial challenges of building and deploying effective retrieval models for semantic matching in real search engine. In this paper, we describe the retrieval system that we developed and deployed in Baidu Search. The system exploits the recent state-of-the-art Chinese pretrained language model, namely Enhanced Representation through kNowledge IntEgration (ERNIE), which facilitates the system with expressive semantic matching. In particular, we developed an ERNIE-based retrieval model, which is equipped with 1) expressive Transformer-based semantic encoders, and 2) a comprehensive multi-stage training paradigm. More importantly, we present a practical system workflow for deploying the model in web-scale retrieval. Eventually, the system is fully deployed into production, where rigorous offline and online experiments were conducted. The results show that the system can perform high-quality candidate retrieval, especially for those tail queries with uncommon demands. Overall, the new retrieval system facilitated by pretrained language model (i.e., ERNIE) can largely improve the usability and applicability of our search engine.
arXiv commentAccepted by KDD 2021
References in corpus (16)
- Distilling the Knowledge in a Neural Network
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- ERNIE: Enhanced Representation through Knowledge Integration
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
- Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
- Neural Interactive Collaborative Filtering
- Semantic Models for the First-stage Retrieval: A Comprehensive Review
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Semantic Modelling with Long-Short-Term Memory for Information Retrieval
- Zero-Resource Translation with Multi-Lingual Neural Machine Translation
- AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
- Neural Passage Retrieval with Improved Negative Contrast
- Memory-efficient Embedding for Recommendations
- Improving Deep Learning For Airbnb Search