Explaining The Efficacy of Counterfactually Augmented Data

Abstract

In attempts to produce ML models less reliant on spurious patterns in NLP datasets, researchers have recently proposed curating counterfactually augmented data (CAD) via a human-in-the-loop process in which given some documents and their (initial) labels, humans must revise the text to make a counterfactual label applicable. Importantly, edits that are not necessary to flip the applicable label are prohibited. Models trained on the augmented data appear, empirically, to rely less on semantically irrelevant words and to generalize better out of domain. While this work draws loosely on causal thinking, the underlying causal model (even at an abstract level) and the principles underlying the observed out-of-domain improvements remain unclear. In this paper, we introduce a toy analog based on linear Gaussian models, observing interesting relationships between causal models, measurement noise, out-of-domain generalization, and reliance on spurious signals. Our analysis provides some insights that help to explain the efficacy of CAD. Moreover, we develop the hypothesis that while adding noise to causal features should degrade both in-domain and out-of-domain performance, adding noise to non-causal features should lead to relative improvements in out-of-domain performance. This idea inspires a speculative test for determining whether a feature attribution technique has identified the causal spans. If adding noise (e.g., by random word flips) to the highlighted spans degrades both in-domain and out-of-domain performance on a battery of challenge datasets, but adding noise to the complement gives improvements out-of-domain, it suggests we have identified causal spans. We present a large-scale empirical study comparing spans edited to create CAD to those selected by attention and saliency maps. Across numerous domains and models, we find that the hypothesized phenomenon is pronounced for CAD.

arXiv commentPublished at ICLR 2021

References in corpus (16)

• Scikit-learn: Machine Learning in Python
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Longformer: The Long-Document Transformer
• A Structured Self-attentive Sentence Embedding
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Invariant Risk Minimization
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Invariant Models for Causal Transfer Learning
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Does mitigating ML's impact disparity require treatment disparity?
• Gender Bias in Neural Natural Language Processing
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?
• Causality in cognitive neuroscience: concepts, challenges, and distributional robustness
• Robustness to Spurious Correlations via Human Annotations
• What do Deep Networks Like to Read?

Cited by in corpus (3)

• A Unifying Causal Framework for Analyzing Dataset Shift-stable Learning Algorithms
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
• A Series of Unfortunate Counterfactual Events: the Role of Time in Counterfactual Explanations