7 citations · 10 across the 20 of their papers we have counts for
6 papers · 1 filter
KronQ: LLM Quantization via Kronecker-Factored Hessian
Donghyun Lee, Yuhang Li, Ruokai Yin +1
Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Most existing second-order PTQ methods, including GP…
Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
Yuhang Li, Donghyun Lee, Ruokai Yin +1
Low-rank decomposition has emerged as an important problem in Large Language Model (LLM) fine-tuning and inference. Through Singular Value Decomposition (SVD), the weight matrix ca…
Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
Ruokai Yin, Sattwik Deb Mishra, Xuan Zuo +3
Distributed LLM inference requires careful coordination of parallelization strategies across hundreds to thousands of NPUs to meet production SLOs. Current systems like Megatron-LM…
DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
Ruokai Yin, Yuhang Li, Donghyun Lee +1
Large language models (LLMs) deliver strong performance but are difficult to deploy due to high memory and compute costs. While pruning reduces these demands, most methods ignore a…
Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
Donghyun Lee, Yuhang Li, Ruokai Yin +2
State Space Models (SSMs) have emerged as powerful alternatives to attention-based Transformers, with Mamba demonstrating impressive efficiency and scalability. As these models gro…
GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration
Yuhang Li, Ruokai Yin, Donghyun Lee +2
We introduce GPTAQ, a novel finetuning-free quantization method for compressing large-scale transformer architectures. Unlike the previous GPTQ method, which independently calibrat…