Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application

Abstract

In e-commerce platforms such as Amazon and TaoBao, ranking items in a search session is a typical multi-step decision-making problem. Learning to rank (LTR) methods have been widely applied to ranking problems. However, such methods often consider different ranking steps in a session to be independent, which conversely may be highly correlated to each other. For better utilizing the correlation between different ranking steps, in this paper, we propose to use reinforcement learning (RL) to learn an optimal ranking policy which maximizes the expected accumulative rewards in a search session. Firstly, we formally define the concept of search session Markov decision process (SSMDP) to formulate the multi-step ranking problem. Secondly, we analyze the property of SSMDP and theoretically prove the necessity of maximizing accumulative rewards. Lastly, we propose a novel policy gradient algorithm for learning an optimal ranking policy, which is able to deal with the problem of high reward variance and unbalanced reward distribution of an SSMDP. Experiments are conducted in simulation and TaoBao search engine. The results demonstrate that our algorithm performs much better than online LTR methods, with more than 40% and 30% growth of total transaction amount in the simulation and the real application, respectively.

References in corpus (4)

• Continuous control with deep reinforcement learning
• Cascading Bandits for Large-Scale Recommendation Problems
• Combinatorial Cascading Bandits
• Online Learning to Rank in Stochastic Click Models

Cited by in corpus (16)

• Deep reinforcement learning for search, recommendation, and online advertising: a survey
• Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
• Self-Supervised Reinforcement Learning for Recommender Systems
• On the Effectiveness of Unlearning in Session-Based Recommendation
• Debiasing Sequential Recommenders through Distributionally Robust Optimization over System Exposure
• Large-scale Interactive Recommendation with Tree-structured Policy Gradient
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• Towards Validating Long-Term User Feedbacks in Interactive Recommendation Systems
• Revenue, Relevance, Arbitrage and More: Joint Optimization Framework for Search Experiences in Two-Sided Marketplaces
• Online Similarity Learning with Feedback for Invoice Line Item Matching
• Corpus-Level End-to-End Exploration for Interactive Systems
• Sequential Search with Off-Policy Reinforcement Learning
• Deep Reinforcement Learning for Personalized Search Story Recommendation
• Top-K Ranking Deep Contextual Bandits for Information Selection Systems
• Learning to Advertise for Organic Traffic Maximization in E-Commerce Product Feeds
• Locality-Sensitive Experience Replay for Online Recommendation