DeepType: Multilingual Entity Linking by Neural Type System Evolution

Abstract

The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data. DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system. First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters. We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.

arXiv commentPresented at AAAI 2018

Cited by in corpus (28)

• Autoregressive Entity Retrieval
• Neural Entity Linking: A Survey of Models Based on Deep Learning
• Falcon 2.0: An Entity and Relation Linking Tool over Wikidata
• Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• Improving Broad-Coverage Medical Entity Linking with Semantic Type Prediction and Large-Scale Datasets
• Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering
• Scalable Zero-shot Entity Linking with Dense Entity Retrieval
• Reddit Entity Linking Dataset
• Entity Linking Meets Deep Learning: Techniques and Solutions
• Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
• Low Resource Recognition and Linking of Biomedical Concepts from a Large Ontology
• Fine-Grained Named Entity Recognition using ELMo and Wikidata
• Fine-Grained Entity Typing for Domain Independent Entity Linking
• Entity Linking via Dual and Cross-Attention Encoders
• Canonicalizing Open Knowledge Bases with Multi-Layered Meta-Graph Neural Network
• CD2CR: Co-reference Resolution Across Documents and Domains
• Neural Network Surgery with Sets
• Hierarchical Entity Typing via Multi-level Learning to Rank
• Encoding Knowledge Graph Entity Aliases in Attentive Neural Network for Wikidata Entity Linking
• AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples
• Improving Distantly-supervised Entity Typing with Compact Latent Space Clustering
• Improving Zero-Shot Multi-Lingual Entity Linking
• Metadata Shaping: Natural Language Annotations for the Tail
• Cross-Lingual Fine-Grained Entity Typing
• Bib2Auth: Deep Learning Approach for Author Disambiguation using Bibliographic Data
• KCAT: A Knowledge-Constraint Typing Annotation Tool
• Interpretable Entity Representations through Large-Scale Typing