62 citations · 205 across the 27 of their papers we have counts for
35 papers
Unsupervised Multimodal Reward Modeling
Yueying Li, Jingxuan Fan, Kianté Brantley +2
Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no hu…
Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Nianli Peng, Geoffrey J. Gordon, Kianté Brantley
Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement up…
Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence
Dries Rooryck, Alex Cai, Yonatan Belinkov +2
Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code…
The State-Prediction Separation Hypothesis
Giovanni Monea, Nathan Godey, Kianté Brantley +1
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction sep…
: Better Prompt Optimization with Fewer Prompts
Zhaolin Gao, Yu, Wang +4
Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what…
Next-token Preference: Scaling Unsupervised Reward Modeling
Jingxuan Fan, Yueying Li, Zhenting Qi +4
Learning from feedback advances the capabilities and safety of frontier models, but preference supervision remains costly and difficult to scale. We introduce next-token preference…
LLMs Can Learn to Reason Via Off-Policy RL
Daniel Ritter, Owen Oertell, Bradley Guo +3
Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training arch…
Expressive Value Learning for Scalable Offline Reinforcement Learning
Nicolas Espinosa-Dice, Kiante Brantley, Wen Sun
Reinforcement learning (RL) is a powerful paradigm for learning to make sequences of decisions. However, RL has yet to be fully leveraged in robotics, principally due to its lack o…
The Emergence of Complex Behavior in Large-Scale Ecological Environments
Joseph Bejjani, Chase Van Amburg, Chengrui Wang +7
We explore how physical scale and population size shape the emergence of complex behaviors in open-ended ecological environments. In our setting, agents are unsupervised and have n…
Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons
Giovanni Monea, Yair Feldman, Shankar Padmanabhan +2
The scalability of large language models for long-context reasoning is severely constrained by the linear growth of their Transformer key-value cache, which incurs significant memo…
Scaling Offline RL via Efficient and Expressive Shortcut Models
Nicolas Espinosa-Dice, Yiyi Zhang, Yiding Chen +5
Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcemen…
Accelerating RL for LLM Reasoning with Optimal Advantage Regression
Kianté Brantley, Mingyu Chen, Zhaolin Gao +4
Reinforcement learning (RL) has emerged as a powerful tool for fine-tuning large language models (LLMs) to improve complex reasoning abilities. However, state-of-the-art policy opt…
Value-Guided Search for Efficient Chain-of-Thought Reasoning
Kaiwen Wang, Jin Peng Zhou, Jonathan Chang +4
In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method doe…
: Provably Optimal Distributional RL for LLM Post-Training
Jin Peng Zhou, Kaiwen Wang, Jonathan Chang +5
Reinforcement learning (RL) post-training is crucial for LLM alignment and reasoning, but existing policy-based methods, such as PPO and DPO, can fall short of fixing shortcuts inh…
Diffusing States and Matching Scores: A New Framework for Imitation Learning
Runzhe Wu, Yiding Chen, Gokul Swamy +2
Adversarial Imitation Learning is traditionally framed as a two-player zero-sum game between a learner and an adversarially chosen cost function, and can therefore be thought of as…
LLMs Are In-Context Bandit Reinforcement Learners
Giovanni Monea, Antoine Bosselut, Kianté Brantley +1
Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context. We investigate a con…
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
Zhaolin Gao, Wenhao Zhan, Jonathan D. Chang +4
Large Language Models (LLMs) have achieved remarkable success at tasks like summarization that involve a single turn of interaction. However, they can still struggle with multi-tur…
Dataset Reset Policy Optimization for RLHF
Jonathan D. Chang, Wenhao Zhan, Owen Oertell +4
Reinforcement Learning (RL) from Human Preference-based feedback is a popular paradigm for fine-tuning generative models, which has produced impressive models such as GPT-4 and Cla…
Adversarial Imitation Learning via Boosting
Jonathan D. Chang, Dhruv Sreenivas, Yingbing Huang +2
Adversarial imitation learning (AIL) has stood out as a dominant framework across various imitation learning (IL) applications, with Discriminator Actor Critic (DAC) (Kostrikov et…
REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan +7
While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) application…
RL for Consistency Models: Faster Reward Guided Text-to-Image Generation
Owen Oertell, Jonathan D. Chang, Yiyi Zhang +2
Reinforcement learning (RL) has improved guided image generation with diffusion models by directly optimizing rewards that capture image quality, aesthetics, and instruction follow…
A Surprising Failure? Multimodal LLMs and the NLVR Challenge
Anne Wu, Kianté Brantley, Yoav Artzi
This study evaluates three state-of-the-art MLLMs -- GPT-4V, Gemini Pro, and the open-source model IDEFICS -- on the compositional natural language vision reasoning task NLVR. Give…
Policy-Gradient Training of Language Models for Ranking
Ge Gao, Jonathan D. Chang, Claire Cardie +2
Text retrieval plays a crucial role in incorporating factual knowledge for decision making into language processing pipelines, ranging from chat-based web search to question answer…
Ranking with Long-Term Constraints
Kianté Brantley, Zhichong Fang, Sarah Dean +1
The feedback that users provide through their choices (e.g., clicks, purchases) is one of the most common types of data readily available for training search and recommendation alg…