8 papers · 1 filter
When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Shuning Shang, Hubert Strauss, Stanley Wei +2
Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely avail…
How Does RL Post-training Induce Skill Composition? A Case Study on Countdown
Simon Park, Simran Kaur, Sanjeev Arora
While reinforcement learning (RL) successfully enhances reasoning in large language models, its role in fostering compositional generalization (the ability to synthesize novel skil…
On the Impossibility of Retrain Equivalence in Machine Unlearning
Jiatong Yu, Yinghui He, Anirudh Goyal +1
Machine unlearning seeks to selectively remove the "influence" of specific training data on a model's outputs. The ideal goal is Retrain Equivalence--behavior identical to a model…
Rethinking Thinking Tokens: LLMs as Improvement Operators
Lovish Madaan, Aniket Didolkar, Suchin Gururangan +6
Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This res…
Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors
Aniket Didolkar, Nicolas Ballas, Sanjeev Arora +1
Large language models (LLMs) now solve multi-step problems by emitting extended chains of thought. During the process, they often re-derive the same intermediate steps across probl…
Weak-to-Strong Generalization Even in Random Feature Networks, Provably
Marko Medvedev, Kaifeng Lyu, Dingli Yu +3
Weak-to-Strong Generalization (Burns et al., 2024) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly ou…