228 citations · 642 across the 22 of their papers we have counts for
24 papers
Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay
Thomas M. McDonald, Lucas Maystre, Mounia Lalmas +2
Recommender systems are a ubiquitous feature of online platforms. Increasingly, they are explicitly tasked with increasing users' long-term satisfaction. In this context, we study…
Deep Reinforcement Learning for Inventory Networks: Toward Reliable Policy Optimization
Matias Alvo, Daniel Russo, Yash Kanoria +1
We argue that inventory management presents unique opportunities for the reliable application of deep reinforcement learning (DRL). To enable this, we emphasize and test two comple…
An Information-Theoretic Analysis of Nonstationary Bandit Learning
Seungki Min, Daniel Russo
In nonstationary bandit learning problems, the decision-maker must continually gather information and adapt their action selection as the latent state of the environment evolves. I…
Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective
Lucas Maystre, Daniel Russo, Yu Zhao
We present a novel podcast recommender system deployed at industrial scale. This system successfully optimizes personal listening journeys that unfold over months for hundreds of m…
On the Statistical Benefits of Temporal Difference Learning
David Cheikhi, Daniel Russo
Given a dataset on actions and resulting long-term rewards, a direct estimation approach fits value functions that minimize prediction error on the training data. Temporal differen…
Adaptive Experimentation in the Presence of Exogenous Nonstationary Variation
Chao Qin, Daniel Russo
We investigate experiments that are designed to select a treatment arm for population deployment. Multi-armed bandit algorithms can enhance efficiency by dynamically allocating mea…
Learning to Stop with Surprisingly Few Samples
Daniel Russo, Assaf Zeevi, Tianyi Zhang
We consider a discounted infinite horizon optimal stopping problem. If the underlying distribution is known a priori, the solution of this problem is obtained via dynamic programmi…
Approximation Benefits of Policy Gradient Methods with Aggregated States
Daniel Russo
Folklore suggests that policy gradient can be more robust to misspecification than its relative, approximate policy iteration. This paper studies the case of state-aggregated repre…
On Linear Convergence of Policy Gradient Methods for Finite MDPs
Jalaj Bhandari, Daniel Russo
We revisit the finite time analysis of policy gradient methods in the one of the simplest settings: finite state and action MDPs with a policy class consisting of all stochastic po…
Policy Gradient Optimization of Thompson Sampling Policies
Seungki Min, Ciamac C. Moallemi, Daniel J. Russo
We study the use of policy gradient algorithms to optimize over a class of generalized Thompson sampling policies. Our central insight is to view the posterior parameter sampled by…
Worst-Case Regret Bounds for Exploration via Randomized Value Functions
Daniel Russo
This paper studies a recent proposal to use randomized value functions to drive exploration in reinforcement learning. These randomized value functions are generated by injecting r…
Global Optimality Guarantees For Policy Gradient Methods
Jalaj Bhandari, Daniel Russo
Policy gradients methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, even…
A Note on the Equivalence of Upper Confidence Bounds and Gittins Indices for Patient Agents
Daniel Russo
This note gives a short, self-contained, proof of a sharp connection between Gittins indices and Bayesian upper confidence bound algorithms. I consider a Gaussian multi-armed bandi…
A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation
Jalaj Bhandari, Daniel Russo, Raghav Singal
Temporal difference learning (TD) is a simple iterative algorithm used to estimate the value function corresponding to a given policy in a Markov decision process. Although TD is o…
Satisficing in Time-Sensitive Bandit Learning
Daniel Russo, Benjamin Van Roy
Much of the recent literature on bandit learning focuses on algorithms that aim to converge on an optimal action. One shortcoming is that this orientation does not account for time…
A Tutorial on Thompson Sampling
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni +2
Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immed…
Improving the Expected Improvement Algorithm
Chao Qin, Diego Klabjan, Daniel Russo
The expected improvement (EI) algorithm is a popular strategy for information collection in optimization under uncertainty. The algorithm is widely known to be too greedy, but neve…
Time-Sensitive Bandit Learning and Satisficing Thompson Sampling
Daniel Russo, David Tse, Benjamin Van Roy
The literature on bandit learning and regret analysis has focused on contexts where the goal is to converge on an optimal action in a manner that limits exploration costs. One shor…
Deep Exploration via Randomized Value Functions
Ian Osband, Benjamin Van Roy, Daniel Russo +1
We study the use of randomized value functions to guide deep exploration in reinforcement learning. This offers an elegant means for synthesizing statistically and computationally…
Simple Bayesian Algorithms for Best Arm Identification
Daniel Russo
This paper considers the optimal adaptive allocation of measurement effort for identifying the best among a finite set of options or designs. An experimenter sequentially chooses d…
Learning to Optimize via Information-Directed Sampling
Daniel Russo, Benjamin Van Roy
We propose information-directed sampling -- a new approach to online optimization problems in which a decision-maker must balance between exploration and exploitation while learnin…
An Information-Theoretic Analysis of Thompson Sampling
Daniel Russo, Benjamin Van Roy
We provide an information-theoretic analysis of Thompson sampling that applies across a broad range of online optimization problems in which a decision-maker must learn from partia…
(More) Efficient Reinforcement Learning via Posterior Sampling
Ian Osband, Daniel Russo, Benjamin Van Roy
Most provably-efficient learning algorithms introduce optimism about poorly-understood states and actions to encourage exploration. We study an alternative approach for efficient e…
Learning to Optimize Via Posterior Sampling
Daniel Russo, Benjamin Van Roy
This paper considers the use of a simple posterior sampling algorithm to balance between exploration and exploitation when learning to optimize actions such as in multi-armed bandi…