Large Language Models as Zero-Shot Conversational Recommenders

Abstract

In this paper, we present empirical studies on conversational recommendation tasks using representative large language models in a zero-shot setting with three primary contributions. (1) Data: To gain insights into model behavior in "in-the-wild" conversational recommendation scenarios, we construct a new dataset of recommendation-related conversations by scraping a popular discussion website. This is the largest public real-world conversational recommendation dataset to date. (2) Evaluation: On the new dataset and two existing conversational recommendation datasets, we observe that even without fine-tuning, large language models can outperform existing fine-tuned conversational recommendation models. (3) Analysis: We propose various probing tasks to investigate the mechanisms behind the remarkable performance of large language models in conversational recommendation. We analyze both the large language models' behaviors and the characteristics of the datasets, providing a holistic understanding of the models' effectiveness, limitations and suggesting directions for the design of future conversational recommenders

arXiv commentAccepted as CIKM 2023 long paper. Longer version is coming soon (e.g., more details about dataset)

References in corpus (36)

• Training language models to follow instructions with human feedback
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Shortcut Learning in Deep Neural Networks
• PaLM: Scaling Language Modeling with Pathways
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• A Survey of Large Language Models
• Scaling Laws for Neural Language Models
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation
• Adversarial Personalized Ranking for Recommendation
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
• Autoregressive Entity Retrieval
• Interactive Path Reasoning on Graph for Conversational Recommendation
• Bias and Debias in Recommender System: A Survey and Future Directions
• Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning
• Transformer Memory as a Differentiable Search Index
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
• User-Centric Conversational Recommendation with Multi-Aspect User Modeling
• What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation
• The False Promise of Imitating Proprietary LLMs
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
• Is ChatGPT a Good Recommender? A Preliminary Study
• Generative Recommendation: Towards Next-generation Recommender Paradigm
• A Survey on Trustworthy Recommender Systems
• M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems
• Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction
• Large Language Models are Zero-Shot Rankers for Recommender Systems
• Leveraging Large Language Models in Conversational Recommender Systems
• Bundle MCR: Towards Conversational Bundle Recommendation
• GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation
• A Next Basket Recommendation Reality Check
• Self-Supervised Bot Play for Conversational Recommendation with Justifications

Cited by in corpus (12)

• Recommender Systems in the Era of Large Language Models (LLMs)
• Large Language Models for Intent-Driven Session Recommendations
• Towards Empathetic Conversational Recommender Systems
• Comprehending Knowledge Graphs with Large Language Models for Recommender Systems
• Adaptive In-Context Learning with Large Language Models for Bundle Generation
• Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation
• LLM-RecG: A Semantic Bias-Aware Framework for Zero-Shot Sequential Recommendation
• Stairway to Fairness: Connecting Group and Individual Fairness
• LARR: Large Language Model Aided Real-time Scene Recommendation with Semantic Understanding
• STEP: Stepwise Curriculum Learning for Context-Knowledge Fusion in Conversational Recommendation
• When the Label Ignores the Request: Auditing Policy-Selected Targets in Synthetic Conversational Music Recommendation
• Improving Conversational Recommendation with Contextual Adaptation of External Recommenders and LLM-based Reranking