The Expressive Power of Word Embeddings

Abstract

We seek to better understand the difference in quality of the several publicly released embeddings. We propose several tasks that help to distinguish the characteristics of different embeddings. Our evaluation of sentiment polarity and synonym/antonym relations shows that embeddings are able to capture surprisingly nuanced semantics even in the absence of sentence structure. Moreover, benchmarking the embeddings shows great variance in quality and characteristics of the semantics captured by the tested embeddings. Finally, we show the impact of varying the number of dimensions and the resolution of each dimension on the effective useful features captured by the embedding space. Our contributions highlight the importance of embeddings for NLP tasks and the effect of their quality on the final results.

submitted to ICML 2013, Deep Learning for Audio, Speech and Language Processing Workshop. 8 pages, 8 figures

References in corpus (1)

• Natural Language Processing (almost) from Scratch

Cited by in corpus (11)

• Polyglot: Distributed Word Representations for Multilingual NLP
• Word Emdeddings through Hellinger PCA
• Legal Document Classification: An Application to Law Area Prediction of Petitions to Public Prosecution Service
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Multichannel Variable-Size Convolution for Sentence Classification
• Freshman or Fresher? Quantifying the Geographic Variation of Internet Language
• Domain Adaptation for Named Entity Recognition in Online Media with Word Embeddings
• Sentiment Analysis of Comments on Rohingya Movement with Support Vector Machine
• A Framework for Evaluation of Machine Reading Comprehension Gold Standards
• Med2Meta: Learning Representations of Medical Concepts with Meta-Embeddings
• Exploring the power of GPU's for training Polyglot language models