Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
arXiv:2109.08331
Abstract
In recommender systems (RecSys) and real-time bidding (RTB) for online advertisements, we often try to optimize sequential decision making using bandit and reinforcement learning (RL) techniques. In these applications, offline reinforcement learning (offline RL) and off-policy evaluation (OPE) are beneficial because they enable safe policy optimization using only logged data without any risky online interaction. In this position paper, we explore the potential of using simulation to accelerate practical research of offline RL and OPE, particularly in RecSys and RTB. Specifically, we discuss how simulation can help us conduct empirical research of offline RL and OPE. We take a position to argue that we should effectively use simulations in the empirical research of offline RL and OPE. To refute the counterclaim that experiments using only real-world data are preferable, we first point out the underlying risks and reproducibility issue in real-world experiments. Then, we describe how these issues can be addressed by using simulations. Moreover, we show how to incorporate the benefits of both real-world and simulation-based experiments to defend our position. Finally, we also present an open challenge to further facilitate practical research of offline RL and OPE in RecSys and RTB, with respect to public simulation platforms. As a possible solution for the issue, we show our ongoing open source project and its potential use case. We believe that building and utilizing simulation-based evaluation platforms for offline RL and OPE will be of great interest and relevance for the RecSys and RTB community.
arXiv commentSimuRec workshop at RecSys2021
References in corpus (22)
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Deep Reinforcement Learning for Page-wise Recommendations
- Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Learning from Logged Implicit Exploration Data
- Deep Reinforcement Learning for List-wise Recommendations
- An Optimistic Perspective on Offline Reinforcement Learning
- Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
- RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising
- COMBO: Conservative Offline Model-Based Policy Optimization
- RecSim: A Configurable Simulation Platform for Recommender Systems
- Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
- Evaluating the Robustness of Off-Policy Evaluation
- NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
- Representation Matters: Offline Pretraining for Sequential Decision Making
- Dynamic Knapsack Optimization Towards Efficient Multi-Channel Sequential Advertising
- DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems
- Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
- Regularized Behavior Value Estimation
- Interpretable performance analysis towards offline reinforcement learning: A dataset perspective