Show Your Work: Improved Reporting of Experimental Results

Abstract

Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.

References in corpus (6)

• Bidirectional Attention Flow for Machine Comprehension
• Enhanced LSTM for Natural Language Inference
• Tune: A Research Platform for Distributed Model Selection and Training
• Deep Reinforcement Learning that Matters
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Cited by in corpus (9)

• When we can trust computers (and when we can't)
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
• The Benchmark Lottery
• Dutch Humor Detection by Generating Negative Examples
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Bridging the Imitation Gap by Adaptive Insubordination
• Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters