Large Language Models as Zero-Shot Conversational Recommenders

Abstract

In this paper, we present empirical studies on conversational recommendation tasks using representative large language models in a zero-shot setting with three primary contributions. (1) Data: To gain insights into model behavior in "in-the-wild" conversational recommendation scenarios, we construct a new dataset of recommendation-related conversations by scraping a popular discussion website. This is the largest public real-world conversational recommendation dataset to date. (2) Evaluation: On the new dataset and two existing conversational recommendation datasets, we observe that even without fine-tuning, large language models can outperform existing fine-tuned conversational recommendation models. (3) Analysis: We propose various probing tasks to investigate the mechanisms behind the remarkable performance of large language models in conversational recommendation. We analyze both the large language models' behaviors and the characteristics of the datasets, providing a holistic understanding of the models' effectiveness, limitations and suggesting directions for the design of future conversational recommenders

Accepted as CIKM 2023 long paper. Longer version is coming soon (e.g., more details about dataset)

References in corpus (16)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Scaling Laws for Neural Language Models
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
• Interactive Path Reasoning on Graph for Conversational Recommendation
• User-Centric Conversational Recommendation with Multi-Aspect User Modeling
• The False Promise of Imitating Proprietary LLMs
• M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems
• Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction
• Leveraging Large Language Models in Conversational Recommender Systems
• Bundle MCR: Towards Conversational Bundle Recommendation
• GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation
• Self-Supervised Bot Play for Conversational Recommendation with Justifications