53 citations · 149 across the 20 of their papers we have counts for
10 papers · 1 filter
Breaking Chains with Trees: Model-Parallel Deep Learning with Time Complexity
Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4
Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become dee…
Probing Length Generalization in Mamba via Image Reconstruction
Jan Rathjens, Robin Schiewer, Laurenz Wiskott +1
Mamba has attracted widespread interest as a general-purpose sequence model due to its low computational complexity and competitive performance relative to transformers. However, i…
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3
In-context learning lets a sequence model adapt to a new task from examples in its input. A prominent line of work shows how self-attention can be constructed to implement gradient…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Cabrel Teguemne Fokam, Marcel Nieveler, Lukas König +3
The increasing size of deep learning models has made distributed training across multiple devices essential. Synchronous, centralized methods incur large communication and synchron…
Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models
Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer +3
Activity and parameter sparsity are two standard methods of making neural networks computationally more efficient. Event-based architectures such as spiking neural networks (SNNs)…
Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models
Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge +3
Event-based sensors are well suited for real-time processing due to their fast response times and encoding of the sensory data as successive temporal differences. These and other v…
Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference
Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer +2
Artificial neural networks open up unprecedented machine learning capabilities at the cost of ever growing computational requirements. Sparsifying the parameters, often achieved th…
Block-local learning with probabilistic latent representations
David Kappel, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam +2
The ubiquitous backpropagation algorithm requires sequential updates through the network introducing a locking problem. In addition, back-propagation relies on the transpose of for…
Efficient Real Time Recurrent Learning through combined activity and parameter sparsity
Anand Subramoney
Backpropagation through time (BPTT) is the standard algorithm for training recurrent neural networks (RNNs), which requires separate simulation phases for the forward and backward…
Efficient recurrent architectures through activity sparsity and sparse back-propagation through time
Anand Subramoney, Khaleelulla Khan Nazeer, Mark Schöne +2
Recurrent neural networks (RNNs) are well suited for solving sequence tasks in resource-constrained systems due to their expressivity and low computational requirements. However, t…