3 citations · 3 across the 7 of their papers we have counts for
8 papers
OASIS: Online Activation Subspace Learning for Memory-Efficient Training
Sakshi Choudhary, Utkarsh Saxena, Kaushik Roy
Training large language models (LLMs) is constrained by memory requirements, with activations accounting for a substantial fraction of the total footprint. Existing approaches redu…
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4
Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval. Continued pretraining on long-context data can help but is…
TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning
Manish Nagaraj, Sakshi Choudhary, Utkarsh Saxena +2
Instruction tuning is essential for aligning large language models (LLMs) to downstream tasks and commonly relies on large, diverse corpora. However, small, high-quality subsets, k…
CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learning
Marco Paul E. Apolinario, Sakshi Choudhary, Kaushik Roy
Continual learning (CL) - the ability to progressively acquire and integrate new concepts - is essential to intelligent systems to adapt to dynamic environments. However, deep neur…
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary +1
Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has be…
SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data
Sakshi Choudhary, Sai Aparna Aketi, Kaushik Roy
Decentralized training enables learning with distributed datasets generated at different locations without relying on a central server. In realistic scenarios, the data distributio…
Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data
Sai Aparna Aketi, Sakshi Choudhary, Kaushik Roy
State-of-the-art decentralized learning algorithms typically require the data distribution to be Independent and Identically Distributed (IID). However, in practical scenarios, the…
CoDeC: Communication-Efficient Decentralized Continual Learning
Sakshi Choudhary, Sai Aparna Aketi, Gobinda Saha +1
Training at the edge utilizes continuously evolving data generated at different locations. Privacy concerns prohibit the co-location of this spatially as well as temporally distrib…