On Design Choices in Similarity-Preserving Sparse Randomized Embeddings

Abstract

Expand & Sparsify is a principle that is observed in anatomically similar neural circuits found in the mushroom body (insects) and the cerebellum (mammals). Sensory data are projected randomly to much higher-dimensionality (expand part) where only few the most strongly excited neurons are activated (sparsify part). This principle has been leveraged to design a FlyHash algorithm that forms similarity-preserving sparse embeddings, which have been found useful for such tasks as novelty detection, pattern recognition, and similarity search. Despite its simplicity, FlyHash has a number of design choices to be set such as preprocessing of the input data, choice of sparsifying activation function, and formation of the random projection matrix. In this paper, we explore the effect of these choices on the performance of similarity search with FlyHash embeddings. We find that the right combination of design choices can lead to drastic difference in the search performance.

8 pages, 4 figures

References in corpus (8)

• Classification using Hyperdimensional Computing: A Review
• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
• A Theoretical Perspective on Hyperdimensional Computing
• A survey of dimensionality reduction techniques based on random projection
• Neural Distributed Autoassociative Memories: A Survey
• Efficient Decoding of Compositional Structure in Holistic Representations
• Streaming Encoding Algorithms for Scalable Hyperdimensional Computing
• Laplace-HDC: Understanding the geometry of binary hyperdimensional computing