Our Evaluation Metric Needs an Update to Encourage Generalization
arXiv:2007.06898
Abstract
Models that surpass human performance on several popular benchmarks display significant degradation in performance on exposure to Out of Distribution (OOD) data. Recent research has shown that models overfit to spurious biases and `hack' datasets, in lieu of learning generalizable features like humans. In order to stop the inflation in model performance -- and thus overestimation in AI systems' capabilities -- we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.
Accepted to ICML UDL 2020
References in corpus (11)
- Distributed Representations of Words and Phrases and their Compositionality
- Gaussian Error Linear Units (GELUs)
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Natural Adversarial Examples
- Adversarial Filters of Dataset Biases
- Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Adversarial NLI: A New Benchmark for Natural Language Understanding
- Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
- DQI: Measuring Data Quality in NLP
- Investigating Biases in Textual Entailment Datasets