activity
20232026
most citedEDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

1 citations · 1 across the 8 of their papers we have counts for

collaborators
Showing 2026Show all

1 paper · 1 filter

cs.LG2026

1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation

Huanxin Sheng, Zhiling Ye, Haonan Wang +3

Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy…