Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond

Abstract

This research aims to explore various methods for assessing user feedback in mixed-initiative conversational search (CS) systems. While CS systems enjoy profuse advancements across multiple aspects, recent research fails to successfully incorporate feedback from the users. One of the main reasons for that is the lack of system-user conversational interaction data. To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems. Specifically, we develop a user simulator, dubbed ConvSim, that, once initialized with an information need description, is capable of providing feedback to a system's responses, as well as answering potential clarifying questions. Our experiments on a wide variety of state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to 16% retrieval performance increase in terms of nDCG@3. Moreover, we observe consistent improvements as the number of feedback rounds increases (35% relative improvement in terms of nDCG@3 after three rounds). This points to a research gap in the development of specific feedback processing modules and opens a potential for significant advancements in CS. To support further research in the topic, we release over 30,000 transcripts of system-simulator interactions based on well-established CS datasets.

11 pages, 2 figures, to be published in SIGIR 2023

References in corpus (11)

• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
• Query Resolution for Conversational Search with Limited Supervision
• Evaluating Conversational Recommender Systems via User Simulation
• Few-Shot Conversational Dense Retrieval
• User Intent Prediction in Information-seeking Conversations
• Evaluating Mixed-initiative Conversational Search Systems via User Simulation
• Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search
• Answer-based Adversarial Training for Generating Clarification Questions
• Analysing Mixed Initiatives and Search Strategies during Conversational Search
• UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
• MIMICS: A Large-Scale Data Collection for Search Clarification