4 papers · 1 filter
From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation
Guobin Shen, Lei Huang, Xiang Cheng +4
On-policy self-distillation has emerged as a promising paradigm for post-training language models, in which the model conditions on environment feedback to serve as its own teacher…
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
Guobin Shen, Xiang Cheng, Chenxiao Zhao +4
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising directi…
One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning
Bowen He, Juncheng Dong, Lin Lin +1
A central challenge in reinforcement learning (RL) is to learn models that generalize beyond the tasks on which they are trained, a goal traditionally pursued through multi-task an…
In-Context Semi-Supervised Learning
Jiashuo Fan, Paul Rosu, Aaron T. Wang +3
There has been significant recent interest in understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicit…