How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

Abstract

Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and activations. Simulations on MNIST and the Google Speech Command Dataset show that such networks demonstrate significantly improved robustness and stability compared to dense networks, while maintaining competitive accuracy. We discuss the potential benefits of sparsity on accuracy, noise robustness, hyperparameter tuning, learning speed, computational efficiency, and power requirements.

arXiv commentReplaced incorrect Fig 5B

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Intriguing properties of neural networks
• Deep Residual Learning for Image Recognition
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Pruning Filters for Efficient ConvNets
• Winner-Take-All Autoencoders
• k-Sparse Autoencoders
• Sparse DNNs with Improved Adversarial Robustness
• How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites
• Sparse Unsupervised Capsules Generalize Better
• Deep Residual Learning for Small-Footprint Keyword Spotting

Cited by in corpus (16)

• A comparison of Vector Symbolic Architectures
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• AHA! an 'Artificial Hippocampal Algorithm' for Episodic Machine Learning
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization
• Algorithmic insights on continual learning from fruit flies
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• Training for temporal sparsity in deep neural networks, application in video processing
• Associative Knowledge Graphs for Efficient Sequence Storage and Retrieval
• A brain basis of dynamical intelligence for AI and computational neuroscience
• Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval
• Spatio-Temporal Sparsification for General Robust Graph Convolution Networks
• HALO: Learning to Prune Neural Networks with Shrinkage
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• Formation of cell assemblies with iterative winners-take-all computation and excitation-inhibition balance