Challenges in Deploying Machine Learning: a Survey of Case Studies

Abstract

In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow we show that practitioners face issues at each stage of the deployment process. The goal of this paper is to lay out a research agenda to explore approaches addressing these challenges.

v3 accepted to publication at ACM Computer Surveys in 2022; v2 presented at The ML-Retrospectives, Surveys & Meta-Analyses Workshop, NeurIPS 2020

References in corpus (12)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Fairness in Machine Learning
• Challenges of Real-World Reinforcement Learning
• "Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment
• The Cost of Training NLP Models: A Concise Overview
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• Denoising Multi-Source Weak Supervision for Neural Text Classification
• Failure Modes in Machine Learning Systems
• Monitoring and explainability of models in production
• Continual Learning in Practice
• Data Engineering for Data Analytics: A Classification of the Issues, and Case Studies
• Towards better data discovery and collection with flow-based programming

Cited by in corpus (38)

• Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation
• Fedstellar: A Platform for Decentralized Federated Learning
• On Leakage in Machine Learning Pipelines
• Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
• Machine Learning (In) Security: A Stream of Problems
• mlpack 4: a fast, header-only C++ machine learning library
• "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
• Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities
• An Empirical Study of Challenges in Machine Learning Asset Management
• Continuous fake media detection: adapting deepfake detectors to new generative techniques
• Maintainability Challenges in ML: A Systematic Literature Review
• Development in times of hype: How freelancers explore Generative AI?
• Making sense of AI systems development
• Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation
• Machine Learning Data Practices through a Data Curation Lens: An Evaluation Framework
• The State of Documentation Practices of Third-party Machine Learning Models and Datasets
• On Sensitivity of Deep Learning Based Text Classification Algorithms to Practical Input Perturbations
• AlerTiger: Deep Learning for AI Model Health Monitoring at LinkedIn
• Being Accountable is Smart: Navigating the Technical and Regulatory Landscape of AI-based Services for Power Grid
• Machine Learning Systems: A Survey from a Data-Oriented Perspective
• Modyn: Data-Centric Machine Learning Pipeline Orchestration
• SecMLOps: A Comprehensive Framework for Integrating Security Throughout the MLOps Lifecycle
• Trustworthy, responsible, ethical AI in manufacturing and supply chains: synthesis and emerging research questions
• ML-EXray: Visibility into ML Deployment on the Edge
• Machine Learning Based on Natural Language Processing to Detect Cardiac Failure in Clinical Narratives
• Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
• How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection
• Towards better data discovery and collection with flow-based programming
• Who is Responsible for Adversarial Defense?
• ALDI++: Automatic and parameter-less discord and outlier detection for building energy load profiles
• We Need to Talk About Data: The Importance of Data Readiness in Natural Language Processing
• Proposing a Framework for Machine Learning Adoption on Legacy Systems
• The Efficiency Misnomer
• Neural forecasting at scale
• Test for non-negligible adverse shifts
• Backward Compatibility in Tree-Based Explanations and Enhanced CART Algorithm
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance
• Knowledge-Adaptation Priors