Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

Hao Wang, Kun Yuan, Wenlin Zhong +4

Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD).…