8 papers
Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
Stanley Wei, Jiin Woo, Kaan Ozkara +1
On-policy distillation (OPD) has recently been introduced as a promising paradigm for continual learning, where a student policy can be supervised at the token-level. Typically, OP…
Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Stanley Wei, Juno Kim
Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at…
When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Shuning Shang, Hubert Strauss, Stanley Wei +2
Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely avail…
Improved high-dimensional estimation with Langevin dynamics and stochastic weight averaging
Stanley Wei, Alex Damian, Jason D. Lee
Significant recent work has studied the ability of gradient descent to recover a hidden planted direction in different high-dimensional settings, including te…
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
Zihan Zheng, Zerui Cheng, Zeyu Shen +16
Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algor…
What Makes a Reward Model a Good Teacher? An Optimization Perspective
Noam Razin, Zixuan Wang, Hubert Strauss +3
The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primarily evaluated through a…
Provable unlearning in topic modeling and downstream tasks
Stanley Wei, Sadhika Malladi, Sanjeev Arora +1
Machine unlearning algorithms are increasingly important as legal concerns arise around the provenance of training data, but verifying the success of unlearning is often difficult.…
Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot
Zixuan Wang, Stanley Wei, Daniel Hsu +1
The transformer architecture has prevailed in various deep learning settings due to its exceptional capabilities to select and compose structural information. Motivated by these ca…