Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning

Abstract

In this paper, we investigate the task of aggregating search results from heterogeneous sources in an E-commerce environment. First, unlike traditional aggregated web search that merely presents multi-sourced results in the first page, this new task may present aggregated results in all pages and has to dynamically decide which source should be presented in the current page. Second, as pointed out by many existing studies, it is not trivial to rank items from heterogeneous sources because the relevance scores from different source systems are not directly comparable. To address these two issues, we decompose the task into two subtasks in a hierarchical structure: a high-level task for source selection where we model the sequential patterns of user behaviors onto aggregated results in different pages so as to understand user intents and select the relevant sources properly; and a low-level task for item presentation where we formulate a slot filling process to sequentially present the items instead of giving each item a relevance score when deciding the presentation order of heterogeneous items. Since both subtasks can be naturally formulated as sequential decision problems and learn from the future user feedback on search results, we build our model with hierarchical reinforcement learning. Extensive experiments demonstrate that our model obtains remarkable improvements in search performance metrics, and achieves a higher user satisfaction.

arXiv commentWWW 19, 11 pages

References in corpus (8)

• Playing Atari with Deep Reinforcement Learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Why We Need New Evaluation Metrics for NLG
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Ranking for Relevance and Display Preferences in Complex Presentation Layouts
• Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning

Cited by in corpus (5)

• Contemporary Recommendation Systems on Big Data and Their Applications: A Survey
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• Generator and Critic: A Deep Reinforcement Learning Approach for Slate Re-ranking in E-commerce
• Spending Money Wisely: Online Electronic Coupon Allocation based on Real-Time User Intent Detection