Controlling Risk of Web Question Answering

Abstract

Web question answering (QA) has become an indispensable component in modern search systems, which can significantly improve users' search experience by providing a direct answer to users' information need. This could be achieved by applying machine reading comprehension (MRC) models over the retrieved passages to extract answers with respect to the search query. With the development of deep learning techniques, state-of-the-art MRC performances have been achieved by recent deep methods. However, existing studies on MRC seldom address the predictive uncertainty issue, i.e., how likely the prediction of an MRC model is wrong, leading to uncontrollable risks in real-world Web QA applications. In this work, we first conduct an in-depth investigation over the risk of Web QA. We then introduce a novel risk control framework, which consists of a qualify model for uncertainty estimation using the probe idea, and a decision model for selectively output. For evaluation, we introduce risk-related metrics, rather than the traditional EM and F1 in MRC, for the evaluation of risk-aware Web QA. The empirical results over both the real-world Web QA dataset and the academic MRC benchmark collection demonstrate the effectiveness of our approach.

arXiv comment42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

References in corpus (17)

• Teaching Machines to Read and Comprehend
• On Calibration of Modern Neural Networks
• Bidirectional Attention Flow for Machine Comprehension
• Machine Comprehension Using Match-LSTM and Answer Pointer
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Predictive Uncertainty Estimation via Prior Networks
• Selective Classification for Deep Neural Networks
• Quasar: Datasets for Question Answering by Search and Reading
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• R${}^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
• Reducing Network Agnostophobia
• Distance-based Confidence Score for Neural Network Classifiers
• Densely Connected Attention Propagation for Reading Comprehension
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• Confidence from Invariance to Image Transformations

Cited by in corpus (3)

• Simulating and Modeling the Risk of Conversational Search
• Less is More: Rejecting Unreliable Reviews for Product Question Answering
• Controlling the Risk of Conversational Search via Reinforcement Learning