Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Abstract

The deployment of machine learning algorithms on resource-constrained edge devices is an important challenge from both theoretical and applied points of view. In this article, we focus on resource-efficient randomly connected neural networks known as Random Vector Functional Link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks. We propose to represent input features via the density-based encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons. Using a collection of 121 real-world datasets from the UCI Machine Learning Repository, we empirically show that the proposed approach demonstrates higher average accuracy than the conventional RVFL. We also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy. In this case, the proposed approach operates only on small n-bits integers, which results in a computationally efficient architecture. Finally, through hardware Field-Programmable Gate Array (FPGA) implementations, we show that such an approach consumes approximately eleven times less energy than that of the conventional RVFL.

7 pages, 7 figures

References in corpus (3)

• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• The High-Dimensional Geometry of Binary Neural Networks
• On the Effectiveness of Discretizing Quantitative Attributes in Linear Classifiers

Cited by in corpus (12)

• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges
• Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
• Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware
• HyperEmbed: Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing enabled Embedding of n-gram Statistics
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• Privacy-Preserving Federated Learning with Differentially Private Hyperdimensional Computing
• Efficient Decoding of Compositional Structure in Holistic Representations
• On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks
• Integer Factorization with Compositional Distributed Representations
• Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing
• Cellular Automata Can Reduce Memory Requirements of Collective-State Computing
• Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks