Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

Abstract

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly ($75\%$) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to $75\%$ reduction in parameter size without significant loss in performance.

ACL 2019

References in corpus (12)

• TensorFlow: A system for large-scale machine learning
• Enhanced LSTM for Natural Language Inference
• A large annotated corpus for learning natural language inference
• Quaternion Knowledge Graph Embeddings
• Deep Complex Networks
• Bilateral Multi-Perspective Matching for Natural Language Sentences
• Quaternion Recurrent Neural Networks
• A Decomposable Attention Model for Natural Language Inference
• Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events
• Attending to Mathematical Language with Transformers
• A Continuously Growing Dataset of Sentential Paraphrases
• Quaternion Collaborative Filtering for Recommendation

Cited by in corpus (3)

• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters