Autoregressive Entity Retrieval

Abstract

Entities are at the center of how we represent and aggregate knowledge. For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per Wikipedia article). The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. Current approaches can be understood as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions. This approach has several shortcomings: (i) context and entity affinity is mainly captured through a vector dot product, potentially missing fine-grained interactions; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time. In this work, we propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion. This mitigates the aforementioned technical issues since: (i) the autoregressive formulation directly captures relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the softmax loss is computed without subsampling negative data. We experiment with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their names. Code and pre-trained models at https://github.com/facebookresearch/GENRE.

arXiv commentAccepted (spotlight) at International Conference on Learning Representations (ICLR) 2021. Code at https://github.com/facebookresearch/GENRE. 20 pages, 9 figures, 8 tables

References in corpus (14)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• Analysis of Named Entity Recognition and Linking for Tweets
• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking
• Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation
• How Context Affects Language Models' Factual Predictions
• DeepType: Multilingual Entity Linking by Neural Type System Evolution
• Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation
• Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search
• Entity-aware ELMo: Learning Contextual Entity Representation for Entity Disambiguation
• Collective Entity Disambiguation with Structured Gradient Tree Boosting

Cited by in corpus (12)

• Neural Entity Linking: A Survey of Models Based on Deep Learning
• Large Language Models as Zero-Shot Conversational Recommenders
• From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer
• GERE: Generative Evidence Retrieval for Fact Verification
• CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
• Continual Learning for Generative Retrieval over Dynamic Corpora
• Entity-aware Transformers for Entity Search
• Entity Linking and Discovery via Arborescence-based Supervised Clustering
• Personal Entity, Concept, and Named Entity Linking in Conversations
• TempEL: Linking Dynamically Evolving and Newly Emerging Entities
• Zero-shot Slot Filling with DPR and RAG
• Evaluating end-to-end entity linking on domain-specific knowledge bases: Learning about ancient technologies from museum collections