Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning

Abstract

Ranking is a fundamental and widely studied problem in scenarios such as search, advertising, and recommendation. However, joint optimization for multi-scenario ranking, which aims to improve the overall performance of several ranking strategies in different scenarios, is rather untouched. Separately optimizing each individual strategy has two limitations. The first one is lack of collaboration between scenarios meaning that each strategy maximizes its own objective but ignores the goals of other strategies, leading to a sub-optimal overall performance. The second limitation is the inability of modeling the correlation between scenarios meaning that independent optimization in one scenario only uses its own user data but ignores the context in other scenarios. In this paper, we formulate multi-scenario ranking as a fully cooperative, partially observable, multi-agent sequential decision problem. We propose a novel model named Multi-Agent Recurrent Deterministic Policy Gradient (MA-RDPG) which has a communication component for passing messages, several private actors (agents) for making actions for ranking, and a centralized critic for evaluating the overall performance of the co-working actors. Each scenario is treated as an agent (actor). Agents collaborate with each other by sharing a global action-value function (the critic) and passing messages that encodes historical information across scenarios. The model is evaluated with online settings on a large E-commerce platform. Results show that the proposed model exhibits significant improvements against baselines in terms of the overall performance.

arXiv commentWWW2018

References in corpus (7)

• Continuous control with deep reinforcement learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Learning Continuous Control Policies by Stochastic Value Gradients
• On Application of Learning to Rank for E-Commerce Search
• Cascade Ranking for Operational E-commerce Search

Cited by in corpus (7)

• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Deep reinforcement learning for search, recommendation, and online advertising: a survey
• Jointly Learning to Recommend and Advertise
• Multi-Task Recommendations with Reinforcement Learning
• Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning
• Multi-Graph based Multi-Scenario Recommendation in Large-scale Online Video Services
• Learning Adaptive Display Exposure for Real-Time Advertising