Analyzing and Characterizing User Intent in Information-seeking Conversations

Abstract

Understanding and characterizing how people interact in information-seeking conversations is crucial in developing conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled dialog dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intent on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.

arXiv commentAccepted by SIGIR 2018 as a short paper

References in corpus (1)

• End-to-end Conversation Modeling Track in DSTC6

Cited by in corpus (27)

• Survey on Evaluation Methods for Dialogue Systems
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• BERT with History Answer Embedding for Conversational Question Answering
• Open-Retrieval Conversational Question Answering
• User Intent Prediction in Information-seeking Conversations
• Conversational Product Search Based on Negative Feedback
• Attentive History Selection for Conversational Question Answering
• Meta-evaluation of Conversational Search Evaluation Metrics
• Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
• Answer Interaction in Non-factoid Question Answering Systems
• User Simulation for Evaluating Information Access Systems
• Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems
• Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset
• Learning to Ask: Conversational Product Search via Representation Learning
• Simulating and Modeling the Risk of Conversational Search
• A Deep Look into Neural Ranking Models for Information Retrieval
• Conversations with Search Engines: SERP-based Conversational Response Generation
• IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking
• Controlling the Risk of Conversational Search via Reinforcement Learning
• Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search
• POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling
• Evaluating Predictive Uncertainty under Distributional Shift on Dialogue Dataset
• Learning to Expand: Reinforced Pseudo-relevance Feedback Selection for Information-seeking Conversations
• Searching Personal Collections