A Systematic Review of Reproducibility Research in Natural Language Processing

Abstract

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

arXiv commentTo be published in proceedings of EACL'21

References in corpus (5)

• Effective LSTMs for Target-Dependent Sentiment Classification
• Sustainable computational science: the ReScience initiative
• Translating from Morphologically Complex Languages: A Paraphrase-Based Approach
• A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Well
• Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis

Cited by in corpus (5)

• Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
• Experimental Standards for Deep Learning in Natural Language Processing Research
• Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspective
• Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study
• How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks