21 citations · 87 across the 19 of their papers we have counts for
20 papers
The Art of Scaling Reinforcement Learning Compute for LLMs
Devvrit Khatri, Lovish Madaan, Rishabh Tiwari +6
Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pr…
SOAP: Improving and Stabilizing Shampoo using Adam
Nikhil Vyas, Depen Morwani, Rosie Zhao +5
There is growing evidence of the effectiveness of Shampoo, a higher-order preconditioning method, over Adam in deep learning optimization tasks. However, Shampoo's drawbacks includ…
Universal Length Generalization with Turing Programs
Kaiying Hou, David Brandfonbrener, Sham Kakade +2
Length generalization refers to the ability to extrapolate from short training sequences to long test sequences and is a challenge for current large language models. While prior wo…
Deconstructing What Makes a Good Optimizer for Language Models
Rosie Zhao, Depen Morwani, David Brandfonbrener +2
Training language models becomes increasingly expensive with scale, prompting numerous attempts to improve optimization efficiency. Despite these efforts, the Adam optimizer remain…
CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
David Brandfonbrener, Hanlin Zhang, Andreas Kirsch +2
Selecting high-quality data for pre-training is crucial in shaping the downstream task performance of language models. A major challenge lies in identifying this optimal subset, a…
Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
Kenneth Li, Samy Jelassi, Hugh Zhang +3
We present an approach called Q-probing to adapt a pre-trained language model to maximize a task-specific reward function. At a high level, Q-probing sits between heavier approache…
VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Search
David Brandfonbrener, Simon Henniger, Sibi Raja +8
Large Language Models (LLMs) can generate useful code, but often the code they generate cannot be trusted to be sound. In this paper, we present VerMCTS, an approach to begin to re…
Repeat After Me: Transformers are Better than State Space Models at Copying
Samy Jelassi, David Brandfonbrener, Sham M. Kakade +1
Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence le…
Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation
David Brandfonbrener, Ofir Nachum, Joan Bruna
In recent years, domains such as natural language processing and image recognition have popularized the paradigm of using large datasets to pretrain representations that can be eff…
Visual Backtracking Teleoperation: A Data Collection Protocol for Offline Image-Based Reinforcement Learning
David Brandfonbrener, Stephen Tu, Avi Singh +4
We consider how to most efficiently leverage teleoperator time to collect data for learning robust image-based value functions and policies for sparse reward robotic tasks. To acco…
Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning
David Brandfonbrener, Remi Tachet des Combes, Romain Laroche
Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that wo…
When does return-conditioned supervised learning work for offline reinforcement learning?
David Brandfonbrener, Alberto Bietti, Jacob Buckman +2
Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL).…
Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
Denis Yarats, David Brandfonbrener, Hao Liu +4
Recent progress in deep learning has relied on access to large and diverse datasets. Such data-driven progress has been less evident in offline reinforcement learning (RL), because…
Quantile Filtered Imitation Learning
David Brandfonbrener, William F. Whitney, Rajesh Ranganath +1
We introduce quantile filtered imitation learning (QFIL), a novel policy improvement operator designed for offline reinforcement learning. QFIL performs policy improvement by runni…
Offline RL Without Off-Policy Evaluation
David Brandfonbrener, William F. Whitney, Rajesh Ranganath +1
Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing…
Evaluating representations by the complexity of learning low-loss predictors
William F. Whitney, Min Jae Song, David Brandfonbrener +2
We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learn…
Offline Contextual Bandits with Overparameterized Models
David Brandfonbrener, William F. Whitney, Rajesh Ranganath +1
Recent results in supervised learning suggest that while overparameterized models have the capacity to overfit, they in fact generalize quite well. We ask whether the same phenomen…
Frequentist Regret Bounds for Randomized Least-Squares Value Iteration
Andrea Zanette, David Brandfonbrener, Emma Brunskill +2
We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning (RL). When the state space is large or continuous, traditional tabular approaches are unfe…
Geometric Insights into the Convergence of Nonlinear TD Learning
David Brandfonbrener, Joan Bruna
While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, an…
Two-vertex generators of Jacobians of graphs
David Brandfonbrener, Pat Devlin, Netanel Friedenberg +4
We give necessary and sufficient conditions under which the Jacobian of a graph is generated by a divisor that is the difference of two vertices. This answers a question posed by B…