53 citations · 149 across the 20 of their papers we have counts for
20 papers
Breaking Chains with Trees: Model-Parallel Deep Learning with Time Complexity
Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4
Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become dee…
Dynamic sparsity in tree-structured feed-forward layers at scale
Reza Sedghi, Robin Schiewer, Anand Subramoney +1
At typical context lengths, the feed-forward MLP block accounts for a large share of a transformer's compute budget, motivating sparse alternatives to dense MLP blocks. We study sp…
Probing Length Generalization in Mamba via Image Reconstruction
Jan Rathjens, Robin Schiewer, Laurenz Wiskott +1
Mamba has attracted widespread interest as a general-purpose sequence model due to its low computational complexity and competitive performance relative to transformers. However, i…
Utilizing dynamic sparsity on pretrained DETR
Reza Sedghi, Anand Subramoney, David Kappel
Efficient inference with transformer-based models remains a challenge, especially in vision tasks like object detection. We analyze the inherent sparsity in the MLP layers of DETR…
STREAM: A Universal State-Space Model for Sparse Geometric Data
Mark Schöne, Yash Bhisikar, Karan Bania +4
Handling sparse and unstructured geometric data, such as point clouds or event-based vision, is a pressing challenge in the field of machine vision. Recently, sequence models such…
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3
In-context learning lets a sequence model adapt to a new task from examples in its input. A prominent line of work shows how self-attention can be constructed to implement gradient…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Cabrel Teguemne Fokam, Marcel Nieveler, Lukas König +3
The increasing size of deep learning models has made distributed training across multiple devices essential. Synchronous, centralized methods incur large communication and synchron…
Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models
Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer +3
Activity and parameter sparsity are two standard methods of making neural networks computationally more efficient. Event-based architectures such as spiking neural networks (SNNs)…
Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models
Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge +3
Event-based sensors are well suited for real-time processing due to their fast response times and encoding of the sensory data as successive temporal differences. These and other v…
Language Modeling on a SpiNNaker 2 Neuromorphic Chip
Khaleelulla Khan Nazeer, Mark Schöne, Rishav Mukherji +4
As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential…
Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference
Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer +2
Artificial neural networks open up unprecedented machine learning capabilities at the cost of ever growing computational requirements. Sparsifying the parameters, often achieved th…
Beyond Weights: Deep learning in Spiking Neural Networks with pure synaptic-delay training
Edoardo W. Grappolini, Anand Subramoney
Biological evidence suggests that adaptation of synaptic delays on short to medium timescales plays an important role in learning in the brain. Inspired by biology, we explore the…
Block-local learning with probabilistic latent representations
David Kappel, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam +2
The ubiquitous backpropagation algorithm requires sequential updates through the network introducing a locking problem. In addition, back-propagation relies on the transpose of for…
Efficient Real Time Recurrent Learning through combined activity and parameter sparsity
Anand Subramoney
Backpropagation through time (BPTT) is the standard algorithm for training recurrent neural networks (RNNs), which requires separate simulation phases for the forward and backward…
Efficient recurrent architectures through activity sparsity and sparse back-propagation through time
Anand Subramoney, Khaleelulla Khan Nazeer, Mark Schöne +2
Recurrent neural networks (RNNs) are well suited for solving sequence tasks in resource-constrained systems due to their expressivity and low computational requirements. However, t…
Exploring hyper-parameter spaces of neuroscience models on high performance computers with Learning to Learn
Alper Yegenoglu, Anand Subramoney, Thorsten Hater +7
Neuroscience models commonly have a high number of degrees of freedom and only specific regions within the parameter space are able to produce dynamics of interest. This makes the…
Embodied Synaptic Plasticity with Online Reinforcement learning
Jacques Kaiser, Michael Hoff, Andreas Konle +8
The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in…
Reservoirs learn to learn
Anand Subramoney, Franz Scherr, Wolfgang Maass
We consider reservoirs in the form of liquid state machines, i.e., recurrently connected networks of spiking neurons with randomly chosen weights. So far only the weights of a line…
Long short-term memory and learning-to-learn in networks of spiking neurons
Guillaume Bellec, Darjan Salaj, Anand Subramoney +2
Recurrent networks of spiking neurons (RSNNs) underlie the astounding computing and learning capabilities of the brain. But computing and learning capabilities of RSNN models have…
Pattern representation and recognition with accelerated analog neuromorphic systems
Mihai A. Petrovici, Sebastian Schmitt, Johann Klähn +33
Despite being originally inspired by the central nervous system, artificial neural networks have diverged from their biological archetypes as they have been remodeled to fit partic…