How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
arXiv:1903.11257
Abstract
Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and activations. Simulations on MNIST and the Google Speech Command Dataset show that such networks demonstrate significantly improved robustness and stability compared to dense networks, while maintaining competitive accuracy. We discuss the potential benefits of sparsity on accuracy, noise robustness, hyperparameter tuning, learning speed, computational efficiency, and power requirements.
Replaced incorrect Fig 5B
References in corpus (3)
Cited by in corpus (9)
- Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
- A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
- Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization
- The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
- Training for temporal sparsity in deep neural networks, application in video processing
- The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
- Spatio-Temporal Sparsification for General Robust Graph Convolution Networks
- A brain basis of dynamical intelligence for AI and computational neuroscience
- Formation of cell assemblies with iterative winners-take-all computation and excitation-inhibition balance