12 papers
FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy
Qingyao Yang, Runming Yang, He Xiao +7
While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specializ…
Simple-OPD: Demystifying Warm-up for On-policy Distillation
Tao Liu, Taiqiang Wu, Mao Zheng +5
On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models, but its effectiveness can depend strongly on the warm-up stage b…
Can We Trust LLMs on Memristors? Diving into Reasoning Ability under Non-Ideality
Taiqiang Wu, Yuxin Cheng, Chenchen Ding +5
Memristor-based analog compute-in-memory (CIM) architectures provide a promising substrate for the efficient deployment of Large Language Models (LLMs), owing to superior energy ef…
ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
Tao Liu, Taiqiang Wu, Runming Yang +3
Supervised fine-tuning (SFT) is a fundamental post-training strategy to align Large Language Models (LLMs) with human intent. However, traditional SFT often ignores the one-to-many…
Revisiting Model Interpolation for Efficient Reasoning
Taiqiang Wu, Runming Yang, Tao Liu +2
Model merging, typically on Instruct and Thinking models, has shown remarkable performance for efficient reasoning. In this paper, we systematically revisit the simplest merging me…
Timber: Training-free Instruct Model Refining with Base via Effective Rank
Taiqiang Wu, Runming Yang, Tao Liu +3
Post-training, which elicits a pretrained Base model into the corresponding Instruct model, is widely considered to be superficial. In this work, we first reinforce this hypothesis…
PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
He Xiao, Runming Yang, Qingyao Yang +6
Post-training quantization (PTQ) of large language models (LLMs) to extremely low bit-widths remains challenging due to the fundamental trade-off between computational efficiency a…
Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
Taiqiang Wu, Runming Yang, Jiayi Li +4
Large language models (LLMs) consistently benefit from further fine-tuning on various tasks. However, we observe that directly tuning the Instruct (i.e., instruction-tuned) models…
Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning
Zhen Li, Yupeng Su, Songmiao Wang +8
Low-bit post-training quantization (PTQ) is a practical route to deploy reasoning-capable LLMs under tight memory and latency budgets, yet it can markedly impair mathematical reaso…
Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning
Zhen Li, Yupeng Su, Runming Yang +5
Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH. However, their substantial computational requirements prese…
LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
Runming Yang, Taiqiang Wu, Jiahao Wang +4
Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstr…
LoCa: Logit Calibration for Knowledge Distillation
Runming Yang, Taiqiang Wu, Yujiu Yang
Knowledge Distillation (KD), aiming to train a better student model by mimicking the teacher model, plays an important role in model compression. One typical way is to align the ou…