Beyond Lexical: A Semantic Retrieval Framework for Textual SearchEngine
arXiv:2008.03917
Abstract
Search engine has become a fundamental component in various web and mobile applications. Retrieving relevant documents from the massive datasets is challenging for a search engine system, especially when faced with verbose or tail queries. In this paper, we explore a vector space search framework for document retrieval. Specifically, we trained a deep semantic matching model so that each query and document can be encoded as a low dimensional embedding. Our model was trained based on BERT architecture. We deployed a fast k-nearest-neighbor index service for online serving. Both offline and online metrics demonstrate that our method improved retrieval performance and search quality considerably, particularly for tail
arXiv comment9 pages
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling
- Query expansion techniques for information retrieval: A survey
- Understanding the Behaviors of BERT in Ranking
- Applying Deep Learning To Airbnb Search
- Neural Models for Information Retrieval
- A Study of MatchPyramid Models on Ad-hoc Retrieval
- Techniques for Deep Query Understanding