168 citations · 361 across the 47 of their papers we have counts for
47 papers
Teaching Arithmetic to Small Transformers
Nayoung Lee, Kartik Sreenivasan, Jason D. Lee +2
Large language models like GPT-4 exhibit emergent capabilities across general-purpose tasks, such as basic arithmetic, when trained on extensive text data, even though these tasks…
Settling the Sample Complexity of Online Reinforcement Learning
Zihan Zhang, Yuxin Chen, Jason D. Lee +1
A central issue lying at the heart of online reinforcement learning (RL) is data efficiency. While a number of recent works achieved asymptotically minimal regret in online RL, the…
Sample Complexity for Quadratic Bandits: Hessian Dependent Bounds and Optimal Algorithms
Qian Yu, Yining Wang, Baihe Huang +2
In stochastic zeroth-order optimization, a problem of practical relevance is understanding how to fully exploit the local geometry of the underlying objective function. We consider…
Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models
Alex Damian, Eshaan Nichani, Rong Ge +1
We focus on the task of learning a single index model with respect to the isotropic Gaussian distribution in dimensions. Prior work has shown that the samp…
Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning
Gen Li, Wenhao Zhan, Jason D. Lee +2
This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment.…
Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning
Yulai Zhao, Zhuoran Yang, Zhaoran Wang +1
Policy optimization methods with function approximation are widely used in multi-agent reinforcement learning. However, it remains elusive how to design such algorithms with statis…
Solving Robust MDPs through No-Regret Dynamics
Etash Kumar Guha
Reinforcement Learning is a powerful framework for training agents to navigate different situations, but it is susceptible to changes in environmental dynamics. However, solving Ma…
Provable Reward-Agnostic Preference-Based Reinforcement Learning
Wenhao Zhan, Masatoshi Uehara, Wen Sun +1
Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than…
Reward Collapse in Aligning Large Language Models
Ziang Song, Tianle Cai, Jason D. Lee +1
The extraordinary capabilities of large language models (LLMs) such as ChatGPT and GPT-4 are in part unleashed by aligning them with reward models that are trained on human prefere…
Fine-Tuning Language Models with Just Forward Passes
Sadhika Malladi, Tianyu Gao, Eshaan Nichani +4
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-…
Provable Offline Preference-Based Reinforcement Learning
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus +2
In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference betwee…
Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability
Jingfeng Wu, Vladimir Braverman, Jason D. Lee
Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen, et al., 2021], where the stepsizes a…
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks
Eshaan Nichani, Alex Damian, Jason D. Lee
One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features. The ability of deep networks to extract salient featur…
Can We Find Nash Equilibria at a Linear Rate in Markov Games?
Zhuoqing Song, Jason D. Lee, Zhuoran Yang
We study decentralized learning in two-player zero-sum discounted Markov games where the goal is to design a policy optimization algorithm for either agent satisfying two propertie…
Provably Efficient Reinforcement Learning via Surprise Bound
Hanlin Zhu, Ruosong Wang, Jason D. Lee
Value function approximation is important in modern reinforcement learning (RL) problems especially when the state space is (infinitely) large. Despite the importance and wide appl…
Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
Masatoshi Uehara, Nathan Kallus, Jason D. Lee +1
In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of…
Looped Transformers as Programmable Computers
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn +3
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punc…
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
Jikai Jin, Zhiyuan Li, Kaifeng Lyu +2
It is believed that Gradient Descent (GD) induces an implicit bias towards good generalization in training machine learning models. This paper provides a fine-grained analysis of t…
Reconstructing Training Data from Model Gradient, Provably
Zihan Wang, Jason D. Lee, Qi Lei
Understanding when and how much a model gradient leaks information about the training sample is an important question in privacy. In this paper, we present a surprising result: eve…
From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent
Satyen Kale, Jason D. Lee, Chris De Sa +2
Stochastic Gradient Descent (SGD) has been the method of choice for learning large-scale non-convex models. While a general analysis of when SGD works has been elusive, there has b…
Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
Alex Damian, Eshaan Nichani, Jason D. Lee
Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness , is bounded by , training is "stable" and the…
PAC Reinforcement Learning for Predictive State Representations
Wenhao Zhan, Masatoshi Uehara, Wen Sun +1
In this paper we study online Reinforcement Learning (RL) in partially observable dynamical systems. We focus on the Predictive State Representations (PSRs) model, which is an expr…
Neural Networks can Learn Representations with Gradient Descent
Alex Damian, Jason D. Lee, Mahdi Soltanolkotabi
Significant theoretical work has established that in specific regimes, neural networks trained by gradient descent behave like kernel methods. However, in practice, it is known tha…
Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
Masatoshi Uehara, Ayush Sekhari, Jason D. Lee +2
We study reinforcement learning with function approximation for large-scale Partially Observable Markov Decision Processes (POMDPs) where the state space and observation space are…