RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising

Abstract

Recommender Systems are becoming ubiquitous in many settings and take many forms, from product recommendation in e-commerce stores, to query suggestions in search engines, to friend recommendation in social networks. Current research directions which are largely based upon supervised learning from historical data appear to be showing diminishing returns with a lot of practitioners report a discrepancy between improvements in offline metrics for supervised learning and the online performance of the newly proposed models. One possible reason is that we are using the wrong paradigm: when looking at the long-term cycle of collecting historical performance data, creating a new version of the recommendation model, A/B testing it and then rolling it out. We see that there a lot of commonalities with the reinforcement learning (RL) setup, where the agent observes the environment and acts upon it in order to change its state towards better states (states with higher rewards). To this end we introduce RecoGym, an RL environment for recommendation, which is defined by a model of user traffic patterns on e-commerce and the users response to recommendations on the publisher websites. We believe that this is an important step forward for the field of recommendation systems research, that could open up an avenue of collaboration between the recommender systems and reinforcement learning communities and lead to better alignment between offline and online performance metrics.

arXiv commentAccepted at the REVEAL workshop at the Twelfth ACM Conference on Recommender Systems (RecSys '18), October 2--7, 2018, Vancouver, BC, Canada

References in corpus (3)

• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Efficient Optimal Learning for Contextual Bandits
• Large-scale Validation of Counterfactual Learning Methods: A Test-Bed

Cited by in corpus (30)

• Deep reinforcement learning for search, recommendation, and online advertising: a survey
• Jointly Learning to Recommend and Advertise
• RecSim: A Configurable Simulation Platform for Recommender Systems
• What are you optimizing for? Aligning Recommender Systems with Human Values
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Self-Supervised Reinforcement Learning for Recommender Systems
• Do Offline Metrics Predict Online Performance in Recommender Systems?
• RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
• Recommendation System-based Upper Confidence Bound for Online Advertising
• Learning from Bandit Feedback: An Overview of the State-of-the-art
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online
• Deep Reinforcement Learning-Based Product Recommender for Online Advertising
• Understanding Longitudinal Dynamics of Recommender Systems with Agent-Based Modeling and Simulation
• T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems
• On the Value of Bandit Feedback for Offline Recommender System Evaluation
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Latent Variable Session-Based Recommendation
• Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning
• No-Regret Algorithms for Private Gaussian Process Bandit Optimization
• How robust is MovieLens? A dataset analysis for recommender systems
• Multiversal Simulacra: Understanding Hypotheticals and Possible Worlds Through Simulation
• Three Methods for Training on Bandit Feedback
• From Clicks to Conversions: Recommendation for long-term reward
• Imitate TheWorld: A Search Engine Simulation Platform
• Locality-Sensitive Experience Replay for Online Recommendation
• Towards Creating a Standardized Collection of Simple and Targeted Experiments to Analyze Core Aspects of the Recommender Systems Problem
• Simulations in Recommender Systems: An industry perspective
• Reconsidering Analytical Variational Bounds for Output Layers of Deep Networks
• Deep Exploration for Recommendation Systems