Uncovering ChatGPT's Capabilities in Recommender Systems

Abstract

The debut of ChatGPT has recently attracted the attention of the natural language processing (NLP) community and beyond. Existing studies have demonstrated that ChatGPT shows significant improvement in a range of downstream NLP tasks, but the capabilities and limitations of ChatGPT in terms of recommendations remain unclear. In this study, we aim to conduct an empirical analysis of ChatGPT's recommendation ability from an Information Retrieval (IR) perspective, including point-wise, pair-wise, and list-wise ranking. To achieve this goal, we re-formulate the above three recommendation policies into a domain-specific prompt format. Through extensive experiments on four datasets from different domains, we demonstrate that ChatGPT outperforms other large language models across all three ranking policies. Based on the analysis of unit cost improvements, we identify that ChatGPT with list-wise ranking achieves the best trade-off between cost and performance compared to point-wise and pair-wise ranking. Moreover, ChatGPT shows the potential for mitigating the cold start problem and explainable recommendation. To facilitate further explorations in this area, the full code and detailed original results are open-sourced at https://github.com/rainym00d/LLM4RS.

arXiv commentAccepted by RecSys 2023

References in corpus (18)

• BPR: Bayesian Personalized Ranking from Implicit Feedback
• PaLM: Scaling Language Modeling with Pathways
• A Survey of Large Language Models
• Scaling Laws for Neural Language Models
• Scaling Instruction-Finetuned Language Models
• Large Language Models are Zero-Shot Reasoners
• How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
• What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation
• A Short Survey of Viewing Large Language Models in Legal Aspect
• ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
• Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams
• Zero-Shot Listwise Document Reranking with a Large Language Model
• SimpleX: A Simple and Strong Baseline for Collaborative Filtering

Cited by in corpus (21)

• Recommender Systems in the Era of Large Language Models (LLMs)
• When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities
• ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Framework
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
• Large Language Models for Intent-Driven Session Recommendations
• Neural Retrievers are Biased Towards LLM-Generated Content
• RecPrompt: A Self-tuning Prompting Framework for News Recommendation Using Large Language Models
• Enhancing Tourism Recommender Systems for Sustainable City Trips Using Retrieval-Augmented Generation
• Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M
• User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendation
• Unleashing the Power of Large Language Model for Denoising Recommendation
• Uncovering Cross-Domain Recommendation Ability of Large Language Models
• RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems
• Adaptive In-Context Learning with Large Language Models for Bundle Generation
• Towards Completeness-Oriented Tool Retrieval for Large Language Models
• Collaboration of Large Language Models and Small Recommendation Models for Device-Cloud Recommendation
• Multi-Layer Ranking with Large Language Models for News Source Recommendation
• Optimizing Recall or Relevance? A Multi-Task Multi-Head Approach for Item-to-Item Retrieval in Recommendation
• MVIGER: Multi-View Variational Integration of Complementary Knowledge for Generative Recommender
• One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation
• Improving Conversational Recommendation with Contextual Adaptation of External Recommenders and LLM-based Reranking