Self-Supervised Reinforcement Learning for Recommender Systems

Abstract

In session-based or sequential recommendation, it is important to consider a number of factors like long-term user engagement, multiple types of user-item interactions such as clicks, purchases etc. The current state-of-the-art supervised approaches fail to model them appropriately. Casting sequential recommendation task as a reinforcement learning (RL) problem is a promising direction. A major component of RL approaches is to train the agent through interactions with the environment. However, it is often problematic to train a recommender in an on-line fashion due to the requirement to expose users to irrelevant recommendations. As a result, learning the policy from logged implicit feedback is of vital importance, which is challenging due to the pure off-policy setting and lack of negative rewards (feedback). In this paper, we propose self-supervised reinforcement learning for sequential recommendation tasks. Our approach augments standard recommendation models with two output layers: one for self-supervised learning and the other for RL. The RL part acts as a regularizer to drive the supervised layer focusing on specific rewards(e.g., recommending items which may lead to purchases rather than clicks) while the self-supervised layer with cross-entropy loss provides strong gradient signals for parameter updates. Based on such an approach, we propose two frameworks namely Self-Supervised Q-learning(SQN) and Self-Supervised Actor-Critic(SAC). We integrate the proposed frameworks with four state-of-the-art recommendation models. Experimental results on two real-world datasets demonstrate the effectiveness of our approach.

SIGIR2020

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Session-based Recommendations with Recurrent Neural Networks
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Generative Adversarial Imitation Learning
• Stabilizing Transformers for Reinforcement Learning
• Generative Adversarial User Model for Reinforcement Learning Based Recommendation System
• Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation
• Self-Attentive Sequential Recommendation
• RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising
• Model-Free Imitation Learning with Policy Optimization
• Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application
• Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems
• Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation
• Exact-K Recommendation via Maximal Clique Optimization

Cited by in corpus (6)

• Disentangling Long and Short-Term Interests for Recommendation
• Self-Supervised Hypergraph Convolutional Networks for Session-based Recommendation
• Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation
• Self-Supervised Graph Co-Training for Session-based Recommendation
• Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
• Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning