1 citations · 1 across the 4 of their papers we have counts for
9 papers
Breaking Chains with Trees: Model-Parallel Deep Learning with Time Complexity
Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4
Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become dee…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Cabrel Teguemne Fokam, Marcel Nieveler, Lukas König +3
The increasing size of deep learning models has made distributed training across multiple devices essential. Synchronous, centralized methods incur large communication and synchron…
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3
Linear recurrent networks (LRNNs) offer linear-time sequence modeling, but standard recurrent updates do not directly expose the supervised products needed for in-context gradient…
Dynamic sparsity in tree-structured feed-forward layers at scale
Reza Sedghi, Robin Schiewer, Anand Subramoney +1
At typical context lengths, the feed-forward MLP block accounts for a large share of a transformer's compute budget, motivating sparse alternatives to dense MLP blocks. We study sp…
Probing Length Generalization in Mamba via Image Reconstruction
Jan Rathjens, Robin Schiewer, Laurenz Wiskott +1
Mamba has attracted widespread interest as a general-purpose sequence model due to its low computational complexity and competitive performance relative to transformers. However, i…
Utilizing dynamic sparsity on pretrained DETR
Reza Sedghi, Anand Subramoney, David Kappel
Efficient inference with transformer-based models remains a challenge, especially in vision tasks like object detection. We analyze the inherent sparsity in the MLP layers of DETR…