Hybrid Policy Distillation for LLMs

Wenhong Zhu, Ruobing Xie, Rui Wang +1

Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimiz…