The emergent algebraic structure of RNNs and embeddings in NLP
arXiv:1803.02839
Abstract
We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.
24 pages, 16 figures
References in corpus (7)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Comparative Study of CNN and RNN for Natural Language Processing
- StarSpace: Embed All The Things!
- Massive Exploration of Neural Machine Translation Architectures
- Learning to Compute Word Embeddings On the Fly
- Monotonic Chunkwise Attention
- Cortical microcircuits as gated-recurrent neural networks