Efficient Decoding of Compositional Structure in Holistic Representations
arXiv:2305.16873 · doi:10.1162/neco_a_01590
Abstract
We investigate the task of retrieving information from compositional distributed representations formed by Hyperdimensional Computing/Vector Symbolic Architectures and present novel techniques which achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, e.g., inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for Hyperdimensional Computing/Vector Symbolic Architectures) are also well-suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks.
arXiv comment28 pages, 5 figures
References in corpus (9)
- Scikit-learn: Machine Learning in Python
- A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
- A Theoretical Perspective on Hyperdimensional Computing
- Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
- Holographic Graph Neuron: a Bio-Inspired Architecture for Pattern Processing
- Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks
- Autoscaling Bloom Filter: Controlling Trade-off Between True and False Positives
- Perceptron Theory Can Predict the Accuracy of Neural Networks
- Orthogonal Matrices for MBAT Vector Symbolic Architectures, and a "Soft" VSA Representation for JSON
Cited by in corpus (4)
- A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
- Perceptron Theory Can Predict the Accuracy of Neural Networks
- Compositional Factorization of Visual Scenes with Convolutional Sparse Coding and Resonator Networks
- On Design Choices in Similarity-Preserving Sparse Randomized Embeddings