2 citations · 2 across the 7 of their papers we have counts for
9 papers
On-Policy Attention Linearization
Arian Raje, Anupam Nayak, Anthony Fei +3
Hybrid transformer architectures that replace most softmax attention layers with linear attention offer transformer-level quality at a fraction of the memory cost. Rather than pret…
PubSwap: Public-Data Off-Policy Coordination for Federated RLVR
Anupam Nayak, Baris Askin, Muhammed Ustaomeroglu +2
Reasoning post-training with reinforcement learning from verifiable rewards (RLVR) is typically studied in centralized settings, yet many realistic applications involve decentraliz…
MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
Arian Raje, Anupam Nayak, Gauri Joshi
Mixture-of-Experts (MoE) model architectures can significantly reduce the number of activated parameters per token, enabling computationally efficient training and inference. Howev…
Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
Arian Raje, Baris Askin, Divyansh Jhunjhunwala +1
Large language models (LLMs) have not yet effectively leveraged the vast amounts of edge-device data, and federated learning (FL) offers a promising paradigm to collaboratively fin…
FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA
Divyansh Jhunjhunwala, Arian Raje, Madan Ravi Ganesh +6
LoRA has emerged as one of the most promising fine-tuning techniques, especially for federated learning (FL), since it significantly reduces communication and computation costs at…
Navigating the Accuracy-Size Trade-Off with Flexible Model Merging
Akash Dhasade, Divyansh Jhunjhunwala, Milos Vujasinovic +2
Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. W…
Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning
Divyansh Jhunjhunwala, Pranay Sharma, Zheng Xu +1
Initializing with pre-trained models when learning on downstream tasks is becoming standard practice in machine learning. Several recent works explore the benefits of pre-trained i…
Erasure Coded Neural Network Inference via Fisher Averaging
Divyansh Jhunjhunwala, Neharika Jali, Gauri Joshi +1
Erasure-coded computing has been successfully used in cloud systems to reduce tail latency caused by factors such as straggling servers and heterogeneous traffic variations. A majo…
FedFisher: Leveraging Fisher Information for One-Shot Federated Learning
Divyansh Jhunjhunwala, Shiqiang Wang, Gauri Joshi
Standard federated learning (FL) algorithms typically require multiple rounds of communication between the server and the clients, which has several drawbacks, including requiring…