Hyperbolic Deep Learning for Chinese Natural Language Understanding
arXiv:1812.10408
Abstract
Recently hyperbolic geometry has proven to be effective in building embeddings that encode hierarchical and entailment information. This makes it particularly suited to modelling the complex asymmetrical relationships between Chinese characters and words. In this paper we first train a large scale hyperboloid skip-gram model on a Chinese corpus, then apply the character embeddings to a downstream hyperbolic Transformer model derived from the principles of gyrovector space for Poincare disk model. In our experiments the character-based Transformer outperformed its word-based Euclidean equivalent. To the best of our knowledge, this is the first time in Chinese NLP that a character-based model outperformed its word-based counterpart, allowing the circumvention of the challenging and domain-dependent task of Chinese Word Segmentation (CWS).
11 pages, 1 figure
References in corpus (9)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Poincaré GloVe: Hyperbolic Word Embeddings
- Hyperbolic Attention Networks
- Hyperbolic Neural Networks
- Skip-gram word embeddings in hyperbolic space
- Gradient descent in hyperbolic space
- Fisher information distance: a geometrical reading
- State-of-the-art Chinese Word Segmentation with Bi-LSTMs