Pitfalls in Machine Learning Research: Reexamining the Development Cycle
arXiv:2011.02832
Abstract
Machine learning has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.
arXiv commentNeurIPS "I Can't Believe It's Not Better!" Workshop
References in corpus (8)
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
- Learning Supervised Topic Models for Classification and Regression from Crowds
- A Step Toward Quantifying Independently Reproducible Machine Learning Research
- Large image datasets: A pyrrhic win for computer vision?
- Machine Learning Explainability for External Stakeholders
- Responses to Critiques on Machine Learning of Criminality Perceptions (Addendum of arXiv:1611.04135)
- MT-Adapted Datasheets for Datasets: Template and Repository
Cited by in corpus (5)
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
- BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
- Cut the CARP: Fishing for zero-shot story evaluation
- Life is not black and white -- Combining Semi-Supervised Learning with fuzzy labels