Out-of-Vocabulary Embedding Imputation with Grounded Language Information by Graph Convolutional Networks
arXiv:1906.03753
Abstract
Due to the ubiquitous use of embeddings as input representations for a wide range of natural language tasks, imputation of embeddings for rare and unseen words is a critical problem in language processing. Embedding imputation involves learning representations for rare or unseen words during the training of an embedding model, often in a post-hoc manner. In this paper, we propose an approach for embedding imputation which uses grounded information in the form of a knowledge graph. This is in contrast to existing approaches which typically make use of vector space properties or subword information. We propose an online method to construct a graph from grounded information and design an algorithm to map from the resulting graphical structure to the space of the pre-trained embeddings. Finally, we evaluate our approach on a range of rare and unseen word tasks across various domains and show that our model can learn better representations. For example, on the Card-660 task our method improves Pearson's and Spearman's correlation coefficients upon the state-of-the-art by 11% and 17.8% respectively using GloVe embeddings.
References in corpus (8)
- Semi-Supervised Classification with Graph Convolutional Networks
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- How Powerful are Graph Neural Networks?
- Question Answering with Subgraph Embeddings
- Learning to Compute Word Embeddings On the Fly
- Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings
- Generalizing Word Embeddings using Bag of Subwords