How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Abstract

Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non--linearly encoded in the embedding space, which questions the cosine-based, unsupervised, evaluation methods. All results and analysis scripts are available online.

References in corpus (5)

• WordRep: A Benchmark for Research on Learning Word Representations
• Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representation on Sequence Labelling Tasks
• Non-distributional Word Vector Representations
• Improving Reliability of Word Similarity Evaluation by Redesigning Annotation Task and Performance Measure
• Expanding Subjective Lexicons for Social Media Mining with Embedding Subspaces

Cited by in corpus (15)

• A Survey of Word Embeddings Evaluation Methods
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Towards Unsupervised Automatic Speech Recognition Trained by Unaligned Speech and Text only
• An Empirical Study on Post-processing Methods for Word Embeddings
• Marked Attribute Bias in Natural Language Inference
• Learning Numeral Embeddings
• RETRO: Relation Retrofitting For In-Database Machine Learning on Textual Data
• DirectProbe: Studying Representations without Classifiers
• Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
• Integrating Lexical Knowledge in Word Embeddings using Sprinkling and Retrofitting
• COSTRA 1.0: A Dataset of Complex Sentence Transformations
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
• Syntree2Vec - An algorithm to augment syntactic hierarchy into word embeddings
• Morphological Skip-Gram: Using morphological knowledge to improve word representation
• An Ensemble Method for Producing Word Representations focusing on the Greek Language