3 papers
cs.CL2026
OPSRD: On-Policy Self-Role Distillation
Weijie Ren, Yanwen Zhang, Hao Li +3
Role prompting elicits specialized behavior from large language models through an expert identity, offering a lightweight way to guide reasoning on demanding tasks. However, evalua…
cs.LG2026
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
Hao Li, MeiJia Chen, Weijie Ren +4
On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. General…
cs.SD2026
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Haojun Zhang, Yi Zou, Min Chen +7
Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause d…