110 citations · 256 across the 18 of their papers we have counts for
19 papers
Muse Glimmer 30B: An On-Device Agentic Model
Arash Ebrahim Abtahizadeh, Muna Aghamelu, Adil Ahmed +327
We introduce Muse Glimmer, an open-source 30B parameter causal language model with perception encoder, purpose-built for autonomous agentic tasks on consumer hardware. Muse Glimmer…
Second-Order Continuum Model for Disordered Traffic
Shashank Rajput, Venkatesan Kanagaraj, Martin Treiber +2
In this study, a two-dimensional second-order macroscopic traffic flow model is proposed to capture the complex dynamics of disordered traffic by explicitly incorporating coupled l…
Inference-Friendly Models With MixAttention
Shashank Rajput, Ying Sheng, Sean Owen +1
The size of the key-value (KV) cache plays a critical role in determining both the maximum context length and the number of concurrent requests supported during inference in modern…
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
Samuel Horvath, Stefanos Laskaridis, Shashank Rajput +1
Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate a…
Recommender Systems with Generative Retrieval
Shashank Rajput, Nikhil Mehta, Anima Singh +10
Modern recommender systems perform large-scale retrieval by first embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search t…
The Expressive Power of Tuning Only the Normalization Layers
Angeliki Giannou, Shashank Rajput, Dimitris Papailiopoulos
Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks. Recent studies on fine-tuning…
Looped Transformers as Programmable Computers
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn +3
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punc…
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
Tuan Dinh, Yuchen Zeng, Ruisu Zhang +6
Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language do…
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment
Tuan Dinh, Jy-yong Sohn, Shashank Rajput +5
Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods. Recent findings have shown that the accuracy and robustness of unsupe…
Finding Everything within Random Binary Networks
Kartik Sreenivasan, Shashank Rajput, Jy-yong Sohn +1
A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achi…
Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond
Chulhee Yun, Shashank Rajput, Suvrit Sra
In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods. Most existing analyses of thes…
An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks
Shashank Rajput, Kartik Sreenivasan, Dimitris Papailiopoulos +1
It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized. Recently, Vershynin (2020) settled a long stan…
Permutation-Based SGD: Is Random Optimal?
Shashank Rajput, Kangwook Lee, Dimitris Papailiopoulos
A recent line of ground-breaking results for permutation-based SGD has corroborated a widely observed phenomenon: random permutations offer faster convergence than with-replacement…
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput +5
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the perf…
Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
Ankit Pensia, Shashank Rajput, Alliot Nagle +2
The strong {\it lottery ticket hypothesis} (LTH) postulates that one can approximate any target neural network by only pruning the weights of a sufficiently over-parameterized rand…
Closing the convergence gap of SGD without replacement
Shashank Rajput, Anant Gupta, Dimitris Papailiopoulos
Stochastic gradient descent without replacement sampling is widely used in practice for model training. However, the vast majority of SGD analyses assumes data is sampled with repl…
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
Shashank Rajput, Hongyi Wang, Zachary Charles +1
To improve the resilience of distributed training to worst-case, or Byzantine node failures, several recent approaches have replaced gradient averaging with robust aggregation meth…
Convergence and Margin of Adversarial Training on Separable Data
Zachary Charles, Shashank Rajput, Stephen Wright +1
Adversarial training is a technique for training robust machine learning models. To encourage robustness, it iteratively computes adversarial examples for the model, and then re-tr…
Does Data Augmentation Lead to Positive Margin?
Shashank Rajput, Zhili Feng, Zachary Charles +2
Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness. DA artificially expands the training set by applying ra…