Jointly Learning to Recommend and Advertise

Abstract

Online recommendation and advertising are two major income channels for online recommendation platforms (e.g. e-commerce and news feed site). However, most platforms optimize recommending and advertising strategies by different teams separately via different techniques, which may lead to suboptimal overall performances. To this end, in this paper, we propose a novel two-level reinforcement learning framework to jointly optimize the recommending and advertising strategies, where the first level generates a list of recommendations to optimize user experience in the long run; then the second level inserts ads into the recommendation list that can balance the immediate advertising revenue from advertisers and the negative influence of ads on long-term user experience. To be specific, the first level tackles high combinatorial action space problem that selects a subset items from the large item space; while the second level determines three internally related tasks, i.e., (i) whether to insert an ad, and if yes, (ii) the optimal ad and (iii) the optimal location to insert. The experimental results based on real-world data demonstrate the effectiveness of the proposed framework. We have released the implementation code to ease reproductivity.

arXiv commentProceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '20), August 23--27, 2020, Virtual Event, CA, USA

References in corpus (23)

• Playing Atari with Deep Reinforcement Learning
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Session-based Recommendations with Recurrent Neural Networks
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Deep Reinforcement Learning for Page-wise Recommendations
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Deep Reinforcement Learning in Large Discrete Action Spaces
• Real-Time Bidding by Reinforcement Learning in Display Advertising
• Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
• Neural Interactive Collaborative Filtering
• Deep Reinforcement Learning for List-wise Recommendations
• Generative Adversarial User Model for Reinforcement Learning Based Recommendation System
• Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising
• Reinforcement Learning based Recommender System using Biclustering Technique
• RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising
• Toward Simulating Environments in Reinforcement Learning Based Recommendations
• Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning
• Top-K Off-Policy Correction for a REINFORCE Recommender System
• Large-scale Interactive Recommendation with Tree-structured Policy Gradient
• A Multi-Agent Reinforcement Learning Method for Impression Allocation in Online Display Advertising
• Solving Continual Combinatorial Selection via Deep Reinforcement Learning
• Adaptive Keywords Extraction with Contextual Bandits for Advertising on Parked Domains
• Learning Adaptive Display Exposure for Real-Time Advertising

Cited by in corpus (14)

• LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems
• Whole-Chain Recommendations
• Sequential Recommendation for Optimizing Both Immediate Feedback and Long-term Retention
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study
• User Response Prediction in Online Advertising
• Hierarchically Constrained Adaptive Ad Exposure in Feeds
• RL-MPCA: A Reinforcement Learning Based Multi-Phase Computation Allocation Approach for Recommender Systems
• Towards Validating Long-Term User Feedbacks in Interactive Recommendation Systems
• AutoLoss: Automated Loss Function Search in Recommendations
• Recommendation Fairness: From Static to Dynamic
• Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems
• Fair Effect Attribution in Parallel Online Experiments
• Locality-Sensitive Experience Replay for Online Recommendation