Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

Miao Rang, Zhenni Bi, Hang Zhou +6

Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, t…