9 papers · 1 filter
JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates
Kai Yi, Vignesh Vivekraja, Harshit Khaitan +1
Quantization-aware training (QAT) is widely deployed but typically relies on the Straight-Through Estimator (STE), which passes gradients through non-differentiable quantizers by f…
WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
Dongyue Li, Zechun Liu, Kai Yi +6
Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is…
Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
Kai Yi
Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a majo…
FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models
Kai Yi, Georg Meinhardt, Laurent Condat +1
Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with…
Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
Georg Meinhardt, Kai Yi, Laurent Condat +1
In Federated Learning (FL), both client resource constraints and communication costs pose major problems for training large models. In the centralized setting, sparse training addr…
Symmetric Pruning of Large Language Models
Kai Yi, Peter Richtárik
Popular post-training pruning methods such as Wanda and RIA are known for their simple, yet effective, designs that have shown exceptional empirical performance. Wanda optimizes pe…