Pretrained Transformers Improve Out-of-Distribution Robustness

Abstract

Although pretrained Transformers such as BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions? We systematically measure out-of-distribution (OOD) generalization for seven NLP datasets by constructing a new robustness benchmark with realistic distribution shifts. We measure the generalization of previous models including bag-of-words models, ConvNets, and LSTMs, and we show that pretrained Transformers' performance declines are substantially smaller. Pretrained transformers are also more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance. We examine which factors affect robustness, finding that larger models are not necessarily more robust, distillation can be harmful, and more diverse pretraining data can enhance robustness. Finally, we show where future work can improve OOD robustness.

arXiv commentACL 2020

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Gaussian Error Linear Units (GELUs)
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Frustratingly Easy Domain Adaptation
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Anomaly Detection with Outlier Exposure
• Pathologies of Neural Models Make Interpretations Difficult
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Natural Adversarial Examples
• Motivating the Rules of the Game for Adversarial Example Research
• Deep Weighted Averaging Classifiers
• A Meta-Analysis of the Anomaly Detection Problem
• Robustness properties of Facebook's ResNeXt WSL models

Cited by in corpus (7)

• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• Uncertainty-Aware Reliable Text Classification
• Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance
• Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning
• On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering
• A Concept Knowledge-Driven Keywords Retrieval Framework for Sponsored Search