Knowing More About Questions Can Help: Improving Calibration in Question Answering

Abstract

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.

arXiv commentACL 2021 (finding)

References in corpus (16)

• Scikit-learn: Machine Learning in Python
• Bidirectional Attention Flow for Machine Comprehension
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Detecting Adversarial Samples from Artifacts
• On Fairness and Calibration
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Understanding Measures of Uncertainty for Adversarial Example Detection
• A BERT Baseline for the Natural Questions
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Self-training Improves Pre-training for Natural Language Understanding
• Calibration of Encoder Decoder Models for Neural Machine Translation
• MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Diverse Ensembles Improve Calibration
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module