1 citations · 1 across the 4 of their papers we have counts for
7 papers
Frequency Domain Reservoir Computing
Klaus Schertler, Xiomara Runge, Andrea Ceni +2
While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balan…
Breaking Chains with Trees: Model-Parallel Deep Learning with Time Complexity
Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4
Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become dee…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Cabrel Teguemne Fokam, Marcel Nieveler, Lukas König +3
The increasing size of deep learning models has made distributed training across multiple devices essential. Synchronous, centralized methods incur large communication and synchron…
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3
Linear recurrent networks (LRNNs) offer linear-time sequence modeling, but standard recurrent updates do not directly expose the supervised products needed for in-context gradient…
Dynamic sparsity in tree-structured feed-forward layers at scale
Reza Sedghi, Robin Schiewer, Anand Subramoney +1
At typical context lengths, the feed-forward MLP block accounts for a large share of a transformer's compute budget, motivating sparse alternatives to dense MLP blocks. We study sp…
Utilizing dynamic sparsity on pretrained DETR
Reza Sedghi, Anand Subramoney, David Kappel
Efficient inference with transformer-based models remains a challenge, especially in vision tasks like object detection. We analyze the inherent sparsity in the MLP layers of DETR…