Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond

Abstract

This research aims to explore various methods for assessing user feedback in mixed-initiative conversational search (CS) systems. While CS systems enjoy profuse advancements across multiple aspects, recent research fails to successfully incorporate feedback from the users. One of the main reasons for that is the lack of system-user conversational interaction data. To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems. Specifically, we develop a user simulator, dubbed ConvSim, that, once initialized with an information need description, is capable of providing feedback to a system's responses, as well as answering potential clarifying questions. Our experiments on a wide variety of state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to 16% retrieval performance increase in terms of nDCG@3. Moreover, we observe consistent improvements as the number of feedback rounds increases (35% relative improvement in terms of nDCG@3 after three rounds). This points to a research gap in the development of specific feedback processing modules and opens a potential for significant advancements in CS. To support further research in the topic, we release over 30,000 transcripts of system-simulator interactions based on well-established CS datasets.

arXiv comment11 pages, 2 figures, to be published in SIGIR 2023

References in corpus (19)

• PaLM: Scaling Language Modeling with Pathways
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Survey on Evaluation Methods for Dialogue Systems
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
• Query Resolution for Conversational Search with Limited Supervision
• Analyzing and Characterizing User Intent in Information-seeking Conversations
• Evaluating Conversational Recommender Systems via User Simulation
• Few-Shot Conversational Dense Retrieval
• User Intent Prediction in Information-seeking Conversations
• Evaluating Mixed-initiative Conversational Search Systems via User Simulation
• Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search
• Answer-based Adversarial Training for Generating Clarification Questions
• Mixed Initiative in Dialogue: An Investigation into Discourse Segmentation
• Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
• Analysing Mixed Initiatives and Search Strategies during Conversational Search
• Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto
• UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
• MIMICS: A Large-Scale Data Collection for Search Clarification
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting

Cited by in corpus (5)

• User Simulation for Evaluating Information Access Systems
• Towards a Formal Characterization of User Simulation Objectives in Conversational Information Access
• GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants
• Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents
• Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search