4 papers
MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation
Jin Cui, Jiaqi Guo, Jiepeng Zhou +6
While Large Language Models (LLMs) have emerged with remarkable capabilities in complex tasks through Chain-of-Thought reasoning, practical resource constraints have sparked intere…
SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
Haiduo Huang, Jiangcheng Song, Yadong Zhang +1
Knowledge distillation (KD) is a standard route to compress Large Language Models (LLMs) into compact students, yet most pipelines uniformly apply token-wise loss regardless of tea…
FastEagle: Cascaded Drafting for Accelerating Speculative Decoding
Haiduo Huang, Jiangcheng Song, Wenzhe Zhao +1
Speculative decoding accelerates generation by drafting candidates and verifying them in parallel, yet state-of-the-art drafters (e.g., EAGLE) still require N sequential passes to…
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Haiduo Huang, Jiangcheng Song, Yadong Zhang +1
Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separat…