Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
arXiv:2101.00288
Abstract
While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts.
arXiv commentACL 2021, main conference, long paper
References in corpus (9)
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Snorkel: Rapid Training Data Creation with Weak Supervision
- Toward Controlled Generation of Text
- It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations
- Unsupervised Text Style Transfer using Language Models as Discriminators
- Yelp Dataset Challenge: Review Rating Prediction
- Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Text
- Counterfactual Explanation Based on Gradual Construction for Deep Networks
- When Choosing Plausible Alternatives, Clever Hans can be Clever
Cited by in corpus (7)
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
- Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing
- Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
- ECINN: Efficient Counterfactuals from Invertible Neural Networks
- Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
- Explaining the Road Not Taken