RLKD: Distilling LLMs' Reasoning via Reinforcement Learning

Shicheng Xu, Liang Pang, Yunchang Zhu +6

Distilling reasoning paths from teacher to student models via supervised fine-tuning (SFT) provides a shortcut for improving the reasoning ability of smaller Large Language Models…