HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning

Abstract

Recent advancements in machine learning research, i.e., deep learning, introduced methods that excel conventional algorithms as well as humans in several complex tasks, ranging from detection of objects in images and speech recognition to playing difficult strategic games. However, the current methodology of machine learning research and consequently, implementations of the real-world applications of such algorithms, seems to have a recurring HARKing (Hypothesizing After the Results are Known) issue. In this work, we elaborate on the algorithmic, economic and social reasons and consequences of this phenomenon. We present examples from current common practices of conducting machine learning research (e.g. avoidance of reporting negative results) and failure of generalization ability of the proposed algorithms and datasets in actual real-life usage. Furthermore, a potential future trajectory of machine learning research and development from the perspective of accountable, unbiased, ethical and privacy-aware algorithmic decision making is discussed. We would like to emphasize that with this discussion we neither claim to provide an exhaustive argumentation nor blame any specific institution or individual on the raised issues. This is simply a discussion put forth by us, insiders of the machine learning field, reflecting on us.

arXiv comment13 pages

References in corpus (19)

• Neural Architecture Search with Reinforcement Learning
• Counterfactual Fairness
• Searching for Activation Functions
• AutoAugment: Learning Augmentation Policies from Data
• Efficient Neural Architecture Search via Parameter Sharing
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Do ImageNet Classifiers Generalize to ImageNet?
• Random Search and Reproducibility for Neural Architecture Search
• Neural Optimizer Search with Reinforcement Learning
• Progressive Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Interpretation of Neural Networks is Fragile
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
• Universality of Deep Convolutional Neural Networks
• Policy Certificates: Towards Accountable Reinforcement Learning
• nGraph-HE: A Graph Compiler for Deep Learning on Homomorphically Encrypted Data
• Expanding search in the space of empirical ML

Cited by in corpus (9)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Evaluation of machine learning algorithms for Health and Wellness applications: a tutorial
• NAS evaluation is frustratingly hard
• Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future
• Bilevel methods for image reconstruction
• Using AntiPatterns to avoid MLOps Mistakes
• Supervised Domain Adaptation using Graph Embedding
• Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search
• Semantic coordinates analysis reveals language changes in the AI field