Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

Leyan Xue, Feng Xiong, Mingjun Ma +1

On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefi…