Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
arXiv:2008.09369
Abstract
With the rise of online e-commerce platforms, more and more customers prefer to shop online. To sell more products, online platforms introduce various modules to recommend items with different properties such as huge discounts. A web page often consists of different independent modules. The ranking policies of these modules are decided by different teams and optimized individually without cooperation, which might result in competition between modules. Thus, the global policy of the whole page could be sub-optimal. In this paper, we propose a novel multi-agent cooperative reinforcement learning approach with the restriction that different modules cannot communicate. Our contributions are three-fold. Firstly, inspired by a solution concept in game theory named correlated equilibrium, we design a signal network to promote cooperation of all modules by generating signals (vectors) for different modules. Secondly, an entropy-regularized version of the signal network is proposed to coordinate agents' exploration of the optimal global policy. Furthermore, experiments based on real-world e-commerce data demonstrate that our algorithm obtains superior performance over baselines.
References in corpus (16)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Counterfactual Multi-Agent Policy Gradients
- Deep Reinforcement Learning for Page-wise Recommendations
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- A Theoretical Analysis of NDCG Type Ranking Measures
- Deep reinforcement learning for search, recommendation, and online advertising: a survey
- Reinforcement Learning based Recommender System using Biclustering Technique
- Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application
- Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions
- Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology
- Ranking for Relevance and Display Preferences in Complex Presentation Layouts
- Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems
- Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning
- Value-aware Recommendation based on Reinforced Profit Maximization in E-commerce Systems