53 citations · 85 across the 35 of their papers we have counts for
28 papers · 1 filter
ExpRL: Exploratory RL for LLM Mid-Training
Violet Xiang, Amrith Setlur, Chase Blagden +2
Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In p…
Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL
Ian Wu, Yuxiao Qu, Amrith Setlur +1
Large Language Models (LLMs) that can continually improve beyond their training budgets are able to solve increasingly difficult problems by adapting at test time, a property we re…
IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL
Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12
While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorl…
Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes
Amrith Setlur, Zijian Wang, Andrew Cohen +2
Typical reinforcement learning (RL) methods for LLM reasoning waste compute on hard problems, where correct on-policy traces are rare, policy gradients vanish, and learning stalls.…
POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration
Yuxiao Qu, Amrith Setlur, Virginia Smith +2
Reinforcement learning (RL) has improved the reasoning abilities of large language models (LLMs), yet state-of-the-art methods still fail to learn on many training problems. On har…
InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning
Matthew Y. R. Yang, Hao Bai, Ian Wu +3
Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs). However, standard RL assigns credit only at…
Lower Bounds for Public-Private Learning under Distribution Shift
Amrith Setlur, Pratiksha Thaker, Jonathan Ullman
The most effective differentially private machine learning algorithms in practice rely on an additional source of purportedly public data. This paradigm is most interesting when th…
e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
Amrith Setlur, Matthew Y. R. Yang, Charlie Snell +5
Test-time scaling offers a promising path to improve LLM reasoning by utilizing more compute at inference time; however, the true promise of this paradigm lies in extrapolation (i.…
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Junhong Shen, Hao Bai, Lunjun Zhang +8
The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this…
Exact Unlearning of Finetuning Data via Model Merging at Scale
Kevin Kuo, Amrith Setlur, Kartik Srinivas +2
Approximate unlearning has gained popularity as an approach to efficiently update an LLM so that it behaves (roughly) as if it was not trained on a subset of data to begin with. Ho…
Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur +4
Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs. Current methods mostly do so via fine-tuning on search traces or ru…
Scaling Test-Time Compute Without Verification or RL is Suboptimal
Amrith Setlur, Nived Rajaraman, Sergey Levine +1
Despite substantial advances in scaling test-time compute, an ongoing debate in the community is how it should be scaled up to enable continued and efficient improvements with scal…
What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?
Katie Kang, Amrith Setlur, Dibya Ghosh +4
Despite the remarkable capabilities of modern large language models (LLMs), the mechanisms behind their problem-solving abilities remain elusive. In this work, we aim to better und…
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
Amrith Setlur, Chirag Nagpal, Adam Fisch +6
A promising approach for improving reasoning in large language models is to use process reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace, po…
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
Amrith Setlur, Saurabh Garg, Xinyang Geng +3
Training on model-generated synthetic data is a promising approach for finetuning LLMs, but it remains unclear when it helps or hurts. In this paper, we investigate this question f…
Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift
Saurabh Garg, Amrith Setlur, Zachary Chase Lipton +3
Self-training and contrastive learning have emerged as leading techniques for incorporating unlabeled data, both under distribution shift (unsupervised domain adaptation) and when…
On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift
Pratiksha Thaker, Amrith Setlur, Zhiwei Steven Wu +1
Public pretraining is a promising approach to improve differentially private model training. However, recent work has noted that many positive research results studying this paradi…
Multitask Learning Can Improve Worst-Group Outcomes
Atharva Kulkarni, Lucio Dery, Amrith Setlur +3
In order to create machine learning systems that serve a variety of users well, it is vital to not only achieve high average performance but also ensure equitable outcomes across d…
Deep Neural Networks Tend To Extrapolate Predictably
Katie Kang, Amrith Setlur, Claire Tomlin +1
Conventional wisdom suggests that neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. Our work reassesses this a…
Contextual Reliability: When Different Features Matter in Different Contexts
Gaurav Ghosal, Amrith Setlur, Daniel S. Brown +2
Deep neural networks often fail catastrophically by relying on spurious correlations. Most prior work assumes a clear dichotomy into spurious and reliable features; however, this i…
Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts
Annie S. Chen, Yoonho Lee, Amrith Setlur +2
Effective machine learning models learn both robust features that directly determine the outcome of interest (e.g., an object with wheels is more likely to be a car), and shortcut…
Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features
Annie S. Chen, Yoonho Lee, Amrith Setlur +2
Transfer learning with a small amount of target data is an effective and common approach to adapting a pre-trained model to distribution shifts. In some situations, target data lab…
Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts
Amrith Setlur, Don Dennis, Benjamin Eysenbach +4
Training machine learning models robust to distribution shifts is critical for real-world applications. Some robust training algorithms (e.g., Group DRO) specialize to group shifts…
Adversarial Unlearning: Reducing Confidence Along Adversarial Directions
Amrith Setlur, Benjamin Eysenbach, Virginia Smith +1
Supervised learning methods trained with maximum likelihood objectives often overfit on training data. Most regularizers that prevent overfitting look to increase confidence on add…