12 papers
Few Bits, One Law: Toward W2A4KV2
Kai Yi, Tarek Elgamal, Sruthikesh Surineni +3
Extreme low-bit LLM compression is most challenging when weights, activations, and KV caches are quantized together: their distributions differ, and quantization errors interact th…
JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates
Kai Yi, Vignesh Vivekraja, Harshit Khaitan +1
Quantization-aware training (QAT) is widely deployed but typically relies on the Straight-Through Estimator (STE), which passes gradients through non-differentiable quantizers by f…
WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
Dongyue Li, Zechun Liu, Kai Yi +6
Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is…
Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
Kai Yi
Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a majo…
All-atom inverse protein folding through discrete flow matching
Kai Yi, Kiarash Jamali, Sjors H. W. Scheres
The recent breakthrough of AlphaFold3 in modeling complex biomolecular interactions, including those between proteins and ligands, nucleotides, or metal ions, creates new opportuni…
Symmetric Pruning of Large Language Models
Kai Yi, Peter Richtárik
Popular post-training pruning methods such as Wanda and RIA are known for their simple, yet effective, designs that have shown exceptional empirical performance. Wanda optimizes pe…
MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects
Lei Fan, Dongdong Fan, Zhiguang Hu +5
We present MANTA, a visual-text anomaly detection dataset for tiny objects. The visual component comprises over 137.3K images across 38 object categories spanning five typical doma…
Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
Kai Yi, Timur Kharisov, Igor Sokolov +1
Virtually all federated learning (FL) methods, including FedAvg, operate in the following manner: i) an orchestrating server sends the current model parameters to a cohort of clien…
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
Vladimir Malinovskii, Denis Mazur, Ivan Ilin +5
There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently o…
Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
Georg Meinhardt, Kai Yi, Laurent Condat +1
In Federated Learning (FL), both client resource constraints and communication costs pose major problems for training large models. In the centralized setting, sparse training addr…
FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity
Kai Yi, Nidham Gazagnadou, Peter Richtárik +1
The interest in federated learning has surged in recent research due to its unique ability to train a global model using privacy-secured information held locally on each client. Th…
FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models
Kai Yi, Georg Meinhardt, Laurent Condat +1
Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with…