Counterfactual Variable Control for Robust and Interpretable Question Answering
arXiv:2010.05581
Abstract
Deep neural network based question answering (QA) models are neither robust nor explainable in many cases. For example, a multiple-choice QA model, tested without any input of question, is surprisingly "capable" to predict the most of correct options. In this paper, we inspect such spurious "capability" of QA models using causal inference. We find the crux is the shortcut correlation, e.g., unrobust word alignment between passage and options learned by the models. We propose a novel approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlation and preserves the comprehensive reasoning for robust QA. Specifically, we leverage multi-branch architecture that allows us to disentangle robust and shortcut correlations in the training process of QA. We then conduct two novel CVC inference methods (on trained models) to capture the effect of comprehensive reasoning as the final prediction. For evaluation, we conduct extensive experiments using two BERT backbones on both multi-choice and span-extraction QA benchmarks. The results show that our CVC achieves high robustness against a variety of adversarial attacks in QA while maintaining good interpretation ability.
References in corpus (17)
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- A large annotated corpus for learning natural language inference
- Adversarial Examples for Evaluating Reading Comprehension Systems
- On Adversarial Examples for Character-Level Neural Machine Translation
- Interventional Few-Shot Learning
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- Causal Intervention for Weakly-Supervised Semantic Segmentation
- Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect
- Unbiased Scene Graph Generation from Biased Training
- Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
- Counterfactual Samples Synthesizing for Robust Visual Question Answering
- Counterfactual VQA: A Cause-Effect Look at Language Bias
- Robust Machine Comprehension Models via Adversarial Training
- Two Causal Principles for Improving Visual Dialog
- QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization