Analogies Explained: Towards Understanding Word Embeddings

Abstract

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "$w_x$ is to $w_y$". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.

Cited by in corpus (14)

• Multi-relational Poincaré Graph Embeddings
• Linguistic Dependencies and Statistical Dependence
• Word Embedding Visualization Via Dictionary Learning
• Cross-Lingual Word Embedding Refinement by ${\ell }_1$ Norm Optimisation
• Interpreting Knowledge Graph Relation Representation from Word Embeddings
• Towards a Flexible Embedding Learning Framework
• The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
• Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction
• InfiniteWalk: Deep Network Embeddings as Laplacian Embeddings with a Nonlinearity
• Soccer Team Vectors
• The Gaussian Transform
• Reflection-based Word Attribute Transfer
• On Dimensional Linguistic Properties of the Word Embedding Space