10 citations · 19 across the 11 of their papers we have counts for
11 papers
Frequency Domain Reservoir Computing
Klaus Schertler, Xiomara Runge, Andrea Ceni +2
While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balan…
Breaking Chains with Trees: Model-Parallel Deep Learning with Time Complexity
Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4
Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become dee…
Dynamic sparsity in tree-structured feed-forward layers at scale
Reza Sedghi, Robin Schiewer, Anand Subramoney +1
At typical context lengths, the feed-forward MLP block accounts for a large share of a transformer's compute budget, motivating sparse alternatives to dense MLP blocks. We study sp…
Utilizing dynamic sparsity on pretrained DETR
Reza Sedghi, Anand Subramoney, David Kappel
Efficient inference with transformer-based models remains a challenge, especially in vision tasks like object detection. We analyze the inherent sparsity in the MLP layers of DETR…
STREAM: A Universal State-Space Model for Sparse Geometric Data
Mark Schöne, Yash Bhisikar, Karan Bania +4
Handling sparse and unstructured geometric data, such as point clouds or event-based vision, is a pressing challenge in the field of machine vision. Recently, sequence models such…
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3
In-context learning lets a sequence model adapt to a new task from examples in its input. A prominent line of work shows how self-attention can be constructed to implement gradient…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Cabrel Teguemne Fokam, Marcel Nieveler, Lukas König +3
The increasing size of deep learning models has made distributed training across multiple devices essential. Synchronous, centralized methods incur large communication and synchron…
Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models
Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer +3
Activity and parameter sparsity are two standard methods of making neural networks computationally more efficient. Event-based architectures such as spiking neural networks (SNNs)…
Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models
Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge +3
Event-based sensors are well suited for real-time processing due to their fast response times and encoding of the sensory data as successive temporal differences. These and other v…
Neuromorphic hardware for sustainable AI data centers
Bernhard Vogginger, Amirhossein Rostami, Vaibhav Jain +9
As humans advance toward a higher level of artificial intelligence, it is always at the cost of escalating computational resource consumption, which requires developing novel solut…
Language Modeling on a SpiNNaker 2 Neuromorphic Chip
Khaleelulla Khan Nazeer, Mark Schöne, Rishav Mukherji +4
As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential…