Hyperbolic Attention Networks
arXiv:1805.09786
Abstract
We introduce hyperbolic attention networks to endow neural networks with enough capacity to match the complexity of data with hierarchical and power-law structure. A few recent approaches have successfully demonstrated the benefits of imposing hyperbolic geometry on the parameters of shallow networks. We extend this line of work by imposing hyperbolic geometry on the activations of neural networks. This allows us to exploit hyperbolic geometry to reason about embeddings produced by deep networks. We achieve this by re-expressing the ubiquitous mechanism of soft attention in terms of operations defined for hyperboloid and Klein models. Our method shows improvements in terms of generalization on neural machine translation, learning on graphs and visual question answering tasks while keeping the neural representations compact.
References in corpus (10)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Semi-Supervised Classification with Graph Convolutional Networks
- Power laws, Pareto distributions and Zipf's law
- Neural Message Passing for Quantum Chemistry
- Spectral Networks and Locally Connected Networks on Graphs
- A simple neural network module for relational reasoning
- Gated Graph Sequence Neural Networks
- Neural Relational Inference for Interacting Systems
- Non-local Neural Networks
- Neural Embeddings of Graphs in Hyperbolic Space
Cited by in corpus (19)
- Adversarial Autoencoders with Constant-Curvature Latent Manifolds
- Discrete and continuous representations and processing in deep learning: Looking forward
- Scalable Hyperbolic Recommender Systems
- Hyperbolic Deep Neural Networks: A Survey
- Latent Variable Modelling with Hyperbolic Normalizing Flows
- Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
- Learning Visual Question Answering by Bootstrapping Hard Attention
- Hyperbolic Interaction Model For Hierarchical Multi-Label Classification
- Mix Dimension in Poincaré Geometry for 3D Skeleton-based Action Recognition
- Hyperbolic Deep Learning for Chinese Natural Language Understanding
- Hyperbolic Neural Collaborative Recommender
- Hyperbolic Neural Networks++
- Lorentzian Graph Convolutional Networks
- HSR: Hyperbolic Social Recommender
- CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Data
- THG: Transformer with Hyperbolic Geometry
- Multi-modal Entity Alignment in Hyperbolic Space
- Nearest Neighbor Search for Hyperbolic Embeddings
- HyperText: Endowing FastText with Hyperbolic Geometry