Word Embeddings: Stability and Semantic Change

Abstract

Word embeddings are computed by a class of techniques within natural language processing (NLP), that create continuous vector representations of words in a language from a large text corpus. The stochastic nature of the training process of most embedding techniques can lead to surprisingly strong instability, i.e. subsequently applying the same technique to the same data twice, can produce entirely different results. In this work, we present an experimental study on the instability of the training process of three of the most influential embedding techniques of the last decade: word2vec, GloVe and fastText. Based on the experimental results, we propose a statistical model to describe the instability of embedding techniques and introduce a novel metric to measure the instability of the representation of an individual word. Finally, we propose a method to minimize the instability - by computing a modified average over multiple runs - and apply it to a specific linguistic problem: The detection and quantification of semantic change, i.e. measuring changes in the meaning and usage of words over time.

References in corpus (9)

• Distributed Representations of Words and Phrases and their Compositionality
• Machine Learning in Automated Text Categorization
• Exploiting Similarities among Languages for Machine Translation
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Diachronic word embeddings and semantic shifts: a survey
• On the Dimensionality of Word Embedding
• Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks
• Understand Functionality and Dimensionality of Vector Embeddings: the Distributional Hypothesis, the Pairwise Inner Product Loss and Its Bias-Variance Trade-off