ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning
arXiv:2206.05240 · doi:10.1145/3534678.3539211
Abstract
Real-Time Bidding (RTB) is an important mechanism in modern online advertising systems. Advertisers employ bidding strategies in RTB to optimize their advertising effects subject to various financial requirements, especially the return-on-investment (ROI) constraint. ROIs change non-monotonically during the sequential bidding process, and often induce a see-saw effect between constraint satisfaction and objective optimization. While some existing approaches show promising results in static or mildly changing ad markets, they fail to generalize to highly dynamic ad markets with ROI constraints, due to their inability to adaptively balance constraints and objectives amidst non-stationarity and partial observability. In this work, we specialize in ROI-Constrained Bidding in non-stationary markets. Based on a Partially Observable Constrained Markov Decision Process, our method exploits an indicator-augmented reward function free of extra trade-off parameters and develops a Curriculum-Guided Bayesian Reinforcement Learning (CBRL) framework to adaptively control the constraint-objective trade-off in non-stationary ad markets. Extensive experiments on a large-scale industrial dataset with two problem settings reveal that CBRL generalizes well in both in-distribution and out-of-distribution data regimes, and enjoys superior learning efficiency and stability.
arXiv commentAccepted by SIGKDD 2022
References in corpus (12)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Reward Constrained Policy Optimization
- (More) Efficient Reinforcement Learning via Posterior Sampling
- Real-Time Bidding by Reinforcement Learning in Display Advertising
- Lyapunov-based Safe Policy Optimization for Continuous Control
- Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising
- Bid Optimization by Multivariable Control in Display Advertising
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Meta reinforcement learning as task inference
- Real-time Bidding for Online Advertising: Measurement and Analysis
- Mechanism Design for Value Maximizers
- Exploration in Online Advertising Systems with Deep Uncertainty-Aware Learning