Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

Abstract

A typical architecture for end-to-end entity linking systems consists of three steps: mention detection, candidate generation and entity disambiguation. In this study we investigate the following questions: (a) Can all those steps be learned jointly with a model for contextualized text-representations, i.e. BERT (Devlin et al., 2019)? (b) How much entity knowledge is already contained in pretrained BERT? (c) Does additional entity knowledge improve BERT's performance in downstream tasks? To this end, we propose an extreme simplification of the entity linking setup that works surprisingly well: simply cast it as a per token classification over the entire entity vocabulary (over 700K classes in our case). We show on an entity linking benchmark that (i) this model improves the entity representations over plain BERT, (ii) that it outperforms entity linking architectures that optimize the tasks separately and (iii) that it only comes second to the current state-of-the-art that does mention detection and entity disambiguation jointly. Additionally, we investigate the usefulness of entity-aware token-representations in the text-understanding benchmark GLUE, as well as the question answering benchmarks SQUAD V2 and SWAG and also the EN-DE WMT14 machine translation benchmark. To our surprise, we find that most of those benchmarks do not benefit from additional entity knowledge, except for a task with very small training data, the RTE task in GLUE, which improves by 2%.

arXiv commentPublished at CoNLL 2019

References in corpus (2)

• Adam: A Method for Stochastic Optimization
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Cited by in corpus (16)

• Autoregressive Entity Retrieval
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Neural Entity Linking: A Survey of Models Based on Deep Learning
• Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
• Entity-aware Transformers for Entity Search
• EntQA: Entity Linking as Question Answering
• Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking
• Reddit Entity Linking Dataset
• Entity Linking Meets Deep Learning: Techniques and Solutions
• Leveraging Encoder-only Large Language Models for Mobile App Review Feature Extraction
• Alquist 3.0: Alexa Prize Bot Using Conversational Knowledge Graph
• Contextualized End-to-End Neural Entity Linking
• Relational World Knowledge Representation in Contextual Language Models: A Review
• Novel Aficionados and Doppelgängers: a referential task for semantic representations of individual entities
• Highly Parallel Autoregressive Entity Linking with Discriminative Correction
• PEL-BERT: A Joint Model for Protocol Entity Linking