Knowing More About Questions Can Help: Improving Calibration in Question Answering

Abstract

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.

ACL 2021 (finding)

References in corpus (9)

• On Fairness and Calibration
• Adversarial Examples for Evaluating Reading Comprehension Systems
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Self-training Improves Pre-training for Natural Language Understanding
• Calibration of Encoder Decoder Models for Neural Machine Translation
• MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Diverse Ensembles Improve Calibration
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module