2 citations · 2 across the 5 of their papers we have counts for
3 papers · 1 filter
SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
Haiduo Huang, Jiangcheng Song, Yadong Zhang +1
Knowledge distillation (KD) is a standard route to compress Large Language Models (LLMs) into compact students, yet most pipelines uniformly apply token-wise loss regardless of tea…
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Haiduo Huang, Jiangcheng Song, Yadong Zhang +1
Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separat…
KernelDNA: Dynamic Kernel Sharing via Decoupled Naive Adapters
Haiduo Huang, Yadong Zhang, Yinghui Xu +1
Dynamic convolution enhances model capacity by adaptively combining multiple kernels, yet faces critical trade-offs: prior works either (1) incur significant parameter overhead by…