Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models

Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer +3

Activity and parameter sparsity are two standard methods of making neural networks computationally more efficient. Event-based architectures such as spiking neural networks (SNNs)…