Csaba Szepesvári

Gaussian-Width Bounds for Offline-to-Online Linear Bandits

Kushagra Chandak, Volodymyr Tkachuk, Xiaoqi Tan +1

We study how offline data can help online learning in stochastic linear bandits. We focus on partially covered offline data, where only $k$ of the $d$ feature directions are poorly…

Foundations of Reinforcement Learning and Control:Connections and New Perspectives

Claire Vernade, Onno Eberhard, Martha White +4

Reinforcement learning and control theory are two adjacent scientific fields that focus on optimizing the controller of unknown dynamical systems using feedback. While both fields…

Randomized Exploration for Linear Bandits via Absolute Perturbations

Toshinori Kitamura, Shuai Liu, Csaba Szepesvári

In stochastic linear bandits, the canonical Upper Confidence Bound (UCB) algorithm admits a simple frequentist regret analysis but can be computationally demanding, while Thompson…

Sharper Guarantees for Misspecified Kernelized Bandit Optimization

Davide Maran, Csaba Szepesvári

Existing guarantees for misspecified kernelized bandit optimization pay for misspecification through kernel complexity: in generic offline bounds, the misspecification level $\vare…

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes

Max Qiushi Lin, Reza Asad, Kevin Tan +3

Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the ex…

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Chang Liu, Yiran Zhao, Lawrence Liu +3

Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long…

Sharp analysis of linear ensemble sampling

David Janz, Arya Akhavan, Csaba Szepesvári

We analyse linear ensemble sampling (ES) with standard Gaussian perturbations in stochastic linear bandits. We show that for ensemble size $m = Θ (d lo g n)$ , ES attains $\tilde O(d^{3…

Learning What to Recommend: Minimax Optimal Simple Regret in Logistic Bandits

Shuai Liu, Alireza Bakhtiari, Alex Ayoub +2

We study stochastic logistic bandits with $d$ -dimensional action features under the simple-regret objective, where a learner uses $T$ rounds of exploration to output a single final…

Eluder dimension: localise it!

Alireza Bakhtiari, Alex Ayoub, Samuel Robertson +2

We establish a lower bound on the eluder dimension of generalised linear model classes, showing that standard eluder dimension-based analysis cannot lead to first-order regret boun…

Rectifying Regression in Reinforcement Learning

Alex Ayoub, David Szepesvári, Alireza Bakhtiari +2

This paper investigates the impact of the loss function in value-based methods for reinforcement learning through an analysis of underlying prediction objectives. We theoretically…

Learning to Reason Efficiently with Discounted Reinforcement Learning

Alex Ayoub, Kavosh Asadi, Dale Schuurmans +2

Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to…

Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear $q^{π}$ -Realizability and Concentrability

Volodymyr Tkachuk, Csaba Szepesvári, Xiaoqi Tan

We study finite-horizon offline reinforcement learning (RL) with function approximation for both policy evaluation and policy optimization. Prior work established that statisticall…

Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits

Max Qiushi Lin, Jincheng Mei, Matin Aghaei +6

Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typica…

Ordering-based Conditions for Global Convergence of Policy Gradient Methods

Jincheng Mei, Bo Dai, Alekh Agarwal +3

We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the pol…

cs.LG2025★ 1 cited

Balancing optimism and pessimism in offline-to-online learning

Flore Sentenac, Ilbin Lee, Csaba Szepesvari

We consider what we call the offline-to-online learning setting, focusing on stochastic finite-armed bandit problems. In offline-to-online learning, a learner starts with offline d…

Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates

Jincheng Mei, Bo Dai, Alekh Agarwal +4

We provide a new understanding of the stochastic gradient bandit algorithm by showing that it converges to a globally optimal policy almost surely using \emph{any} constant learnin…

Almost Free: Self-concordance in Natural Exponential Families and an Application to Bandits

Shuai Liu, Alex Ayoub, Flore Sentenac +2

We prove that single-parameter natural exponential families with subexponential tails are self-concordant with polynomial-sized parameters. For subgaussian natural exponential fami…

cs.LG2024★ 3 cited

To Believe or Not to Believe Your LLM

Yasin Abbasi Yadkori, Ilja Kuzborskij, András György +1

We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider bo…

Confident Natural Policy Gradient for Local Planning in $q_{π}$ -realizable Constrained MDPs

Tian Tian, Lin F. Yang, Csaba Szepesvári

The constrained Markov decision process (CMDP) framework emerges as an important reinforcement learning approach for imposing safety or other critical objectives while maximizing c…

Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear $q^{π}$ -Realizability and Concentrability

Volodymyr Tkachuk, Gellért Weisz, Csaba Szepesvári

We consider offline reinforcement learning (RL) in $H$ -horizon Markov decision processes (MDPs) under the linear $q^{π}$ -realizability assumption, where the action-value function of…

cs.LG2024★ 2 cited

Mitigating LLM Hallucinations via Conformal Abstention

Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz +9

We develop a principled procedure for determining when a large language model (LLM) should abstain from responding (e.g., by saying "I don't know") in a general domain, instead of…

Regret Minimization via Saddle Point Optimization

Johannes Kirschner, Seyed Alireza Bakhtiari, Kushagra Chandak +2

A long line of works characterizes the sample complexity of regret minimization in sequential decision-making by min-max programs. In the corresponding saddle-point game, the min-p…