Probabilistic FastText for Multi-Sense Word Embeddings

Abstract

We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share statistical strength across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, the proposed model is the first to achieve multi-sense representations while having enriched semantics on rare words.

arXiv commentPublished at ACL 2018

References in corpus (2)

• Efficient Estimation of Word Representations in Vector Space
• Character-Aware Neural Language Models

Cited by in corpus (11)

• Modeling Islamist Extremist Communications on Social Media using Contextual Dimensions: Religion, Ideology, and Hate
• Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
• Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning
• EDS-MEMBED: Multi-sense embeddings based on enhanced distributional semantic structures via a graph walk over word senses
• Hit ratio: An Evaluation Metric for Hashtag Recommendation
• Improving Document Classification with Multi-Sense Embeddings
• Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
• Sequence-to-Set Semantic Tagging: End-to-End Multi-label Prediction using Neural Attention for Complex Query Reformulation and Automated Text Categorization
• SLAM-Inspired Simultaneous Contextualization and Interpreting for Incremental Conversation Sentences
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
• A novel approach to sentiment analysis in Persian using discourse and external semantic information