A Multi-Resolution Word Embedding for Document Retrieval from Large Unstructured Knowledge Bases
arXiv:1902.00663
Abstract
Deep language models learning a hierarchical representation proved to be a powerful tool for natural language processing, text mining and information retrieval. However, representations that perform well for retrieval must capture semantic meaning at different levels of abstraction or context-scopes. In this paper, we propose a new method to generate multi-resolution word embeddings that represent documents at multiple resolutions in terms of context-scopes. In order to investigate its performance,we use the Stanford Question Answering Dataset (SQuAD) and the Question Answering by Search And Reading (QUASAR) in an open-domain question-answering setting, where the first task is to find documents useful for answering a given question. To this end, we first compare the quality of various text-embedding methods for retrieval performance and give an extensive empirical comparison with the performance of various non-augmented base embeddings with and without multi-resolution representation. We argue that multi-resolution word embeddings are consistently superior to the original counterparts and deep residual neural models specifically trained for retrieval purposes can yield further significant gains when they are used for augmenting those embeddings.
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Deep Learning Face Representation by Joint Identification-Verification
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- Quasar: Datasets for Question Answering by Search and Reading
- Evaluation of sentence embeddings in downstream and linguistic probing tasks
- Learning Semantic Similarity for Very Short Texts
- Learning Fine-grained Image Similarity with Deep Ranking
- Reading Wikipedia to Answer Open-Domain Questions
- Text Embeddings for Retrieval From a Large Knowledge Base