2 papers
cs.LG2026
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
Zhu Zhang, Jixun Wang, Xiaoang Xu +6
On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher sup…
cs.CL2026
Dual Dimensionality for Local and Global Attention
Zhiyuan Wang, Xuan Luo, Sirui Zeng +1
Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its dista…