PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions

Abstract

Hypercomplex neural networks have proven to reduce the overall number of parameters while ensuring valuable performance by leveraging the properties of Clifford algebras. Recently, hypercomplex linear layers have been further improved by involving efficient parameterized Kronecker products. In this paper, we define the parameterization of hypercomplex convolutional layers and introduce the family of parameterized hypercomplex neural networks (PHNNs) that are lightweight and efficient large-scale models. Our method grasps the convolution rules and the filter organization directly from data without requiring a rigidly predefined domain structure to follow. PHNNs are flexible to operate in any user-defined or tuned domain, from 1D to $n$D regardless of whether the algebra rules are preset. Such a malleability allows processing multidimensional inputs in their natural domain without annexing further dimensions, as done, instead, in quaternion neural networks for 3D inputs like color images. As a result, the proposed family of PHNNs operates with $1/n$ free parameters as regards its analog in the real domain. We demonstrate the versatility of this approach to multiple domains of application by performing experiments on various image datasets as well as audio datasets in which our method outperforms real and quaternion-valued counterparts. Full code is available at: https://github.com/eleGAN23/HyperNets.

arXiv commentSubmitted to IEEE Transactions on Neural Networks and Learning Systems

References in corpus (12)

• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks
• Sound Event Detection: A Tutorial
• Quaternion Convolutional Neural Networks for Heterogeneous Image Processing
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events
• On the Dynamics of Hopfield Neural Networks on Unit Quaternions
• A Broad Class of Discrete-Time Hypercomplex-Valued Hopfield Neural Networks
• Quaternion Generative Adversarial Networks
• Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
• Hypercomplex-Valued Recurrent Correlation Neural Networks
• Quaternion-Valued Recurrent Projection Neural Networks on Unit Quaternions

Cited by in corpus (4)

• Learning Speech Emotion Representations in the Quaternion Domain
• Hypercomplex Image-to-Image Translation
• Understanding Vector-Valued Neural Networks and Their Relationship with Real and Hypercomplex-Valued Neural Networks
• KHNNs: hypercomplex neural networks computations via Keras using TensorFlow and PyTorch