Towards Amortized Ranking-Critical Training for Collaborative Filtering

Abstract

Collaborative filtering is widely used in modern recommender systems. Recent research shows that variational autoencoders (VAEs) yield state-of-the-art performance by integrating flexible representations from deep neural networks into latent variable models, mitigating limitations of traditional linear factor models. VAEs are typically trained by maximizing the likelihood (MLE) of users interacting with ground-truth items. While simple and often effective, MLE-based training does not directly maximize the recommendation-quality metrics one typically cares about, such as top-N ranking. In this paper we investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to directly optimize the non-differentiable quality metrics of interest. Specifically, we train a critic network to approximate ranking-based metrics, and then update the actor network (represented here by a VAE) to directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require to re-run the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists. Empirically, we show that the proposed methods outperform several state-of-the-art baselines, including recently-proposed deep learning approaches, on three large-scale real-world datasets. The code to reproduce the experimental results and figure plots is on Github: https://github.com/samlobel/RaCT_CF

arXiv commentThe first two authors contributed equally to this manuscript. Code: https://github.com/samlobel/RaCT_CF

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Sequence Level Training with Recurrent Neural Networks
• cGANs with Projection Discriminator
• Deep Reinforcement Learning for Dialogue Generation
• Adversarial Personalized Ranking for Recommendation
• Embarrassingly Shallow Autoencoders for Sparse Data
• An Actor-Critic Algorithm for Sequence Prediction
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Adversarial Learning for Neural Dialogue Generation
• A Neural Autoregressive Approach to Collaborative Filtering
• Adversarial Ranking for Language Generation
• Metadata Embeddings for User and Item Cold-start Recommendations
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Amortized Inference Regularization
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• Outer Product-based Neural Collaborative Filtering