Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters

Abstract

Recent works have demonstrated reasonable success of representation learning in hypercomplex space. Specifically, "fully-connected layers with Quaternions" (4D hypercomplex numbers), which replace real-valued matrix multiplications in fully-connected layers with Hamilton products of Quaternions, both enjoy parameter savings with only 1/4 learnable parameters and achieve comparable performance in various applications. However, one key caveat is that hypercomplex space only exists at very few predefined dimensions (4D, 8D, and 16D). This restricts the flexibility of models that leverage hypercomplex multiplications. To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predefined. As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary nD hypercomplex space, providing more architectural flexibility using arbitrarily $1/n$ learnable parameters compared with the fully-connected layer counterpart. Experiments of applications to the LSTM and Transformer models on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural flexibility and effectiveness of the proposed approach.

Published as a conference paper at the 9th International Conference on Learning Representations (ICLR 2021)

References in corpus (4)

• Quaternion Convolutional Neural Networks for Heterogeneous Image Processing
• Quaternion Recurrent Neural Networks
• Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
• Neural Networks with Complex and Quaternion Inputs

Cited by in corpus (8)

• A Theoretical Perspective on Hyperdimensional Computing
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Hypercomplex Image-to-Image Translation
• Understanding Vector-Valued Neural Networks and Their Relationship with Real and Hypercomplex-Valued Neural Networks
• Convolutional Hypercomplex Embeddings for Link Prediction
• HopfE: Knowledge Graph Representation Learning using Inverse Hopf Fibrations
• Efficient Attribute Injection for Pretrained Language Models