Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Distillation as Probability Transport: Routed On-Policy Distillation
Tianle Xia, Lingxiang Hu, Yiding Sun +6
On-policy distillation (OPD) transfers teacher knowledge on student-generated trajectories, but efficient sampled objectives reduce the teacher distribution to scalar credit on ind…
cs.LG2024
An Integrated Data Processing Framework for Pretraining Foundation Models
Yiding Sun, Feng Wang, Yutao Zhu +2
The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners ofte…