The Scientific Method in the Science of Machine Learning

Abstract

In the quest to align deep learning with the sciences to address calls for rigor, safety, and interpretability in machine learning systems, this contribution identifies key missing pieces: the stages of hypothesis formulation and testing, as well as statistical and systematic uncertainty estimation -- core tenets of the scientific method. This position paper discusses the ways in which contemporary science is conducted in other domains and identifies potentially useful practices. We present a case study from physics and describe how this field has promoted rigor through specific methodological practices, and provide recommendations on how machine learning researchers can adopt these practices into the research ecosystem. We argue that both domain-driven experiments and application-agnostic questions of the inner workings of fundamental building blocks of machine learning models ought to be examined with the tools of the scientific method, to ensure we not only understand effect, but also begin to understand cause, which is the raison d'être of science.

4 pages + 1 appendix. Presented at the ICLR 2019 Debugging Machine Learning Models workshop

References in corpus (2)

• The RooFit toolkit for data modeling
• Avoiding a Tragedy of the Commons in the Peer Review Process

Cited by in corpus (5)

• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging
• Showing Your Work Doesn't Always Work
• Semantic coordinates analysis reveals language changes in the AI field
• Expected Validation Performance and Estimation of a Random Variable's Maximum