Learning to Compute Word Embeddings On the Fly

Abstract

Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.

References in corpus (9)

• Dynamic Coattention Networks For Question Answering
• A large annotated corpus for learning natural language inference
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
• Blocks and Fuel: Frameworks for deep learning
• Dynamic Integration of Background Knowledge in Neural NLU Systems
• Vocabulary Selection Strategies for Neural Machine Translation
• A Comparative Study of Word Embeddings for Reading Comprehension
• Vocabulary Manipulation for Neural Machine Translation
• Leveraging Lexical Resources for Learning Entity Embeddings in Multi-Relational Data

Cited by in corpus (21)

• BosphorusSign22k Sign Language Recognition Dataset
• The Effect of Natural Distribution Shift on Question Answering Models
• xSense: Learning Sense-Separated Sparse Representations and Textual Definitions for Explainable Word Sense Networks
• Enriching Rare Word Representations in Neural Language Models by Embedding Matrix Augmentation
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• Reasoning over RDF Knowledge Bases using Deep Learning
• Making Neural Machine Reading Comprehension Faster
• Latent Relation Language Models
• Taking Notes on the Fly Helps BERT Pre-training
• Improving Pre-Trained Multilingual Models with Vocabulary Expansion
• A Hybrid Semantic Parsing Approach for Tabular Data Analysis
• Multi-channel Reverse Dictionary Model
• "What Are You Trying to Do?" Semantic Typing of Event Processes
• Conditional Generators of Words Definitions
• Updater-Extractor Architecture for Inductive World State Representations
• Out-of-Vocabulary Embedding Imputation with Grounded Language Information by Graph Convolutional Networks
• Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters
• Knowledge Efficient Deep Learning for Natural Language Processing
• The emergent algebraic structure of RNNs and embeddings in NLP
• Interactive Re-Fitting as a Technique for Improving Word Embeddings
• Automatic Construction of Sememe Knowledge Bases via Dictionaries