62 citations · 205 across the 27 of their papers we have counts for
7 papers · 1 filter
Unsupervised Multimodal Reward Modeling
Yueying Li, Jingxuan Fan, Kianté Brantley +2
Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no hu…
Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Nianli Peng, Geoffrey J. Gordon, Kianté Brantley
Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement up…
Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence
Dries Rooryck, Alex Cai, Yonatan Belinkov +2
Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code…
The State-Prediction Separation Hypothesis
Giovanni Monea, Nathan Godey, Kianté Brantley +1
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction sep…
: Better Prompt Optimization with Fewer Prompts
Zhaolin Gao, Yu, Wang +4
Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what…
Next-token Preference: Scaling Unsupervised Reward Modeling
Jingxuan Fan, Yueying Li, Zhenting Qi +4
Learning from feedback advances the capabilities and safety of frontier models, but preference supervision remains costly and difficult to scale. We introduce next-token preference…
LLMs Can Learn to Reason Via Off-Policy RL
Daniel Ritter, Owen Oertell, Bradley Guo +3
Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training arch…