Skip-gram word embeddings in hyperbolic space

Abstract

Recent work has demonstrated that embeddings of tree-like graphs in hyperbolic space surpass their Euclidean counterparts in performance by a large margin. Inspired by these results and scale-free structure in the word co-occurrence graph, we present an algorithm for learning word embeddings in hyperbolic space from free text. An objective function based on the hyperbolic distance is derived and included in the skip-gram negative-sampling architecture of word2vec. The hyperbolic word embeddings are then evaluated on word similarity and analogy benchmarks. The results demonstrate the potential of hyperbolic word embeddings, particularly in low dimensions, though without clear superiority over their Euclidean counterparts. We further discuss subtleties in the formulation of the analogy task in curved spaces.

References in corpus (10)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Hyperbolic Geometry of Complex Networks
• Representation Tradeoffs for Hyperbolic Embeddings
• Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry
• Neural Embeddings of Graphs in Hyperbolic Space
• Hyperbolic Neural Networks
• Gradient descent in hyperbolic space
• Large-Margin Classification in Hyperbolic Space

Cited by in corpus (8)

• Poincaré GloVe: Hyperbolic Word Embeddings
• Scalable Hyperbolic Recommender Systems
• Hyperbolic Interaction Model For Hierarchical Multi-Label Classification
• Incorporating Domain Knowledge into Health Recommender Systems using Hyperbolic Embeddings
• Hyperbolic Deep Learning for Chinese Natural Language Understanding
• Representation Learning with Weighted Inner Product for Universal Approximation of General Similarities
• Augmenting the User-Item Graph with Textual Similarity Models
• Learning phylogenetic trees as hyperbolic point configurations