End-to-End Self-Debiasing Framework for Robust NLU Training

Abstract

Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performance on out-of-distribution (OOD) ones. We introduce a simple yet effective debiasing framework whereby the shallow representations of the main model are used to derive a bias model and both models are trained simultaneously. We demonstrate on three well studied NLU tasks that despite its simplicity, our method leads to competitive OOD results. It significantly outperforms other debiasing approaches on two tasks, while still delivering high in-distribution performance.

Findings ACL 2021

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Learning from others' mistakes: Avoiding dataset biases without modeling them
• Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures

Cited by in corpus (4)

• How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
• Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding
• Towards Zero-Shot Knowledge Distillation for Natural Language Processing
• Knowledge Distillation with Noisy Labels for Natural Language Understanding