1 citations · 1 across the 3 of their papers we have counts for
5 papers · 1 filter
On-Policy Attention Linearization
Arian Raje, Anupam Nayak, Anthony Fei +3
Hybrid transformer architectures that replace most softmax attention layers with linear attention offer transformer-level quality at a fraction of the memory cost. Rather than pret…
MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
Arian Raje, Anupam Nayak, Gauri Joshi
Mixture-of-Experts (MoE) model architectures can significantly reduce the number of activated parameters per token, enabling computationally efficient training and inference. Howev…
Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
Arian Raje, Baris Askin, Divyansh Jhunjhunwala +1
Large language models (LLMs) have not yet effectively leveraged the vast amounts of edge-device data, and federated learning (FL) offers a promising paradigm to collaboratively fin…
FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA
Divyansh Jhunjhunwala, Arian Raje, Madan Ravi Ganesh +6
LoRA has emerged as one of the most promising fine-tuning techniques, especially for federated learning (FL), since it significantly reduces communication and computation costs at…
Federated LoRA with Sparse Communication
Kevin Kuo, Arian Raje, Kousik Rajesh +1
Low-rank adaptation (LoRA) is a natural method for finetuning in communication-constrained machine learning settings such as cross-device federated learning. Prior work that has st…