Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Weak-to-Strong On-Policy Distillation
Fangxu Yu, Weijia Xu, Michael Xu +2
On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabiliti…
cs.LG2025
TRA: Better Length Generalisation with Threshold Relative Attention
Mattia Opper, Roland Fernandez, Paul Smolensky +1
Transformers struggle with length generalisation, displaying poor performance even on basic tasks. We test whether these limitations can be explained through two key failures of th…