Poincaré GloVe: Hyperbolic Word Embeddings

Abstract

Words are not created equal. In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal. In this paper, justified by the notion of delta-hyperbolicity or tree-likeliness of a space, we propose to embed words in a Cartesian product of hyperbolic spaces which we theoretically connect to the Gaussian word embeddings and their Fisher geometry. This connection allows us to introduce a novel principled hypernymy score for word embeddings. Moreover, we adapt the well-known Glove algorithm to learn unsupervised word embeddings in this type of Riemannian manifolds. We further explain how to solve the analogy task using the Riemannian parallel transport that generalizes vector arithmetics to this new type of geometry. Empirically, based on extensive experiments, we prove that our embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection. In particular, for word hypernymy, we obtain new state-of-the-art on fully unsupervised WBLESS classification accuracy.

References in corpus (4)

• Efficient Estimation of Word Representations in Vector Space
• Riemannian Adaptive Optimization Methods
• Generalizing Point Embeddings using the Wasserstein Space of Elliptical Distributions
• Skip-gram word embeddings in hyperbolic space

Cited by in corpus (36)

• Network Geometry
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Hyperspherical Prototype Networks
• Discriminative Topic Mining via Category-Name Guided Text Embedding
• Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders
• ACE-HGNN: Adaptive Curvature Exploration Hyperbolic Graph Neural Network
• Constant Curvature Graph Convolutional Networks
• Multi-relational Poincaré Graph Embeddings
• HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems
• Hyperbolic Geometric Graph Representation Learning for Hierarchy-imbalance Node Classification
• HyperKG: Hyperbolic Knowledge Graph Embeddings for Knowledge Base Completion
• Hyperbolic Node Embedding for Signed Networks
• Knowledge Graph Embeddings and Explainable AI
• Hyperbolic Interaction Model For Hierarchical Multi-Label Classification
• Robust Large-Margin Learning in Hyperbolic Space
• Hyperbolic Variational Graph Neural Network for Modeling Dynamic Graphs
• Fully Hyperbolic Rotation for Knowledge Graph Embedding
• Mix Dimension in Poincaré Geometry for 3D Skeleton-based Action Recognition
• APo-VAE: Text Generation in Hyperbolic Space
• Hyperbolic Deep Learning for Chinese Natural Language Understanding
• Hyperbolic Neural Collaborative Recommender
• Hyperbolic Generative Adversarial Network
• Cross-lingual Entity Alignment with Incidental Supervision
• ArGoT: A Glossary of Terms extracted from the arXiv
• Hyperbolic Neural Networks++
• Lorentzian Graph Convolutional Networks
• Differentiating through the Fréchet Mean
• HSR: Hyperbolic Social Recommender
• Hyperbolic Manifold Regression
• Nearest Neighbor Search for Hyperbolic Embeddings
• Exploring Semantic Capacity of Terms
• Adaptive Hierarchical Similarity Metric Learning with Noisy Labels
• Overlapping Spaces for Compact Graph Representations
• HyperText: Endowing FastText with Hyperbolic Geometry
• Identifying and Mitigating Gender Bias in Hyperbolic Word Embeddings
• Adaptive versus Standard Descent Methods and Robustness Against Adversarial Examples