Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning

Abstract

Sponsored search is an indispensable business model and a major revenue contributor of almost all the search engines. From the advertisers' side, participating in ranking the search results by paying for the sponsored search advertisement to attract more awareness and purchase facilitates their commercial goal. From the users' side, presenting personalized advertisement reflecting their propensity would make their online search experience more satisfactory. Sponsored search platforms rank the advertisements by a ranking function to determine the list of advertisements to show and the charging price for the advertisers. Hence, it is crucial to find a good ranking function which can simultaneously satisfy the platform, the users and the advertisers. Moreover, advertisements showing positions under different queries from different users may associate with advertisement candidates of different bid price distributions and click probability distributions, which requires the ranking functions to be optimized adaptively to the traffic characteristics. In this work, we proposed a generic framework to optimize the ranking functions by deep reinforcement learning methods. The framework is composed of two parts: an offline learning part which initializes the ranking functions by learning from a simulated advertising environment, allowing adequate exploration of the ranking function parameter space without hurting the performance of the commercial platform. An online learning part which further optimizes the ranking functions by adapting to the online data distribution. Experimental results on a large-scale sponsored search platform confirm the effectiveness of the proposed method.

arXiv commentrevise some content

References in corpus (9)

• Continuous control with deep reinforcement learning
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• A User Simulator for Task-Completion Dialogues
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• Off-Policy Actor-Critic
• Budget Optimization for Sponsored Search: Censored Learning in MDPs