8 papers · 1 filter
Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
Zixuan Xie, Xinyu Liu, Claire Chen +3
In-context reinforcement learning (ICRL) studies agents that, after pretraining, adapt to new tasks by conditioning on additional context without parameter updates. Existing theore…
Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
Zixuan Xie, Xinyu Liu, Rohan Chandra +1
In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context.…
Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Drift
Xinyu Liu, Zixuan Xie, Shangtong Zhang
Establishing almost sure convergence rates for stochastic approximation and reinforcement learning under Markovian noise is a fundamental theoretical challenge. We make progress to…
Pure Exploration via Frank-Wolfe Self-Play
Xinyu Liu, Chao Qin, Wei You
We study pure exploration in structured stochastic multi-armed bandits, aiming to efficiently identify the correct hypothesis from a finite set of alternatives. For a broad class o…
Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning
Xinyu Liu, Zixuan Xie, Shangtong Zhang
The Robbins-Siegmund theorem establishes the convergence of stochastic processes that are almost supermartingales and is one of the most commonly used approaches for analyzing stoc…
Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features
Zixuan Xie, Xinyu Liu, Rohan Chandra +1
Linear TD() is one of the most fundamental reinforcement learning algorithms for policy evaluation. Previously, convergence rates are typically established under the assumption…
Linear -Learning Does Not Diverge in : Convergence Rates to a Bounded Set
Xinyu Liu, Zixuan Xie, Shangtong Zhang
-learning is one of the most fundamental reinforcement learning algorithms. It is widely believed that -learning with linear function approximation (i.e., linear -learning…
Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise
Xiaochi Qian, Zixuan Xie, Xinyu Liu +1
This paper establishes the first almost sure convergence rate and the first maximal concentration bound with exponential tails for general contractive stochastic approximation algo…