7 papers · 1 filter
OASIS: Online Activation Subspace Learning for Memory-Efficient Training
Sakshi Choudhary, Utkarsh Saxena, Kaushik Roy
Training large language models (LLMs) is constrained by memory requirements, with activations accounting for a substantial fraction of the total footprint. Existing approaches redu…
KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
Utkarsh Saxena, Kaushik Roy
Quantizing the key-value (KV) cache is a promising strategy for improving the inference efficiency of large language models (LLMs). However, aggressive quantization to very low pre…
ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Utkarsh Saxena, Sayeh Sharify, Kaushik Roy +1
Post-training quantization (PTQ) of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. Quantization of all weight, act…
SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data
Sakshi Choudhary, Sai Aparna Aketi, Kaushik Roy
Decentralized training enables learning with distributed datasets generated at different locations without relying on a central server. In realistic scenarios, the data distributio…
AdaGossip: Adaptive Consensus Step-size for Decentralized Deep Learning with Communication Compression
Sai Aparna Aketi, Abolfazl Hashemi, Kaushik Roy
Decentralized learning is crucial in supporting on-device learning over large distributed datasets, eliminating the need for a central server. However, the communication overhead r…
Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data
Sai Aparna Aketi, Sakshi Choudhary, Kaushik Roy
State-of-the-art decentralized learning algorithms typically require the data distribution to be Independent and Identically Distributed (IID). However, in practical scenarios, the…
Cross-feature Contrastive Loss for Decentralized Deep Learning on Heterogeneous Data
Sai Aparna Aketi, Kaushik Roy
The current state-of-the-art decentralized learning algorithms mostly assume the data distribution to be Independent and Identically Distributed (IID). However, in practical scenar…