A Lyapunov Analysis of Softmax Policy Gradient for Stochastic Bandits

Abstract

We adapt the analysis of policy gradient for continuous time $k$-armed stochastic bandits by Lattimore (2026) to the standard discrete time setup. As in continuous time, we prove that with learning rate $\hat{I}\cdot =O(\hat{I}{”}_{\min }^2/(\hat{I}{”}_{\max }\log (n)))$ the regret is $O(k\log (k)\log (n)/\hat{I}\cdot )$ where $n$ is the horizon and $\hat{I}{”}_{\min }$ and $\hat{I}{”}_{\max }$ are the minimum and maximum gaps.

6 pages