"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning

Abstract

Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationalization of machine learning, or MLOps, requires MLEs to have proficiency in data science and engineering. When considered holistically, the job seems staggering -- how do MLEs do MLOps, and what are their unaddressed challenges? To address these questions, we conducted semi-structured ethnographic interviews with 18 MLEs working on various applications, including chatbots, autonomous vehicles, and finance. We find that MLEs engage in a workflow of (i) data preparation, (ii) experimentation, (iii) evaluation throughout a multi-staged deployment, and (iv) continual monitoring and response. Throughout this workflow, MLEs collaborate extensively with data scientists, product stakeholders, and one another, supplementing routine verbal exchanges with communication tools ranging from Slack to organization-wide ticketing and reporting systems. We introduce the 3Vs of MLOps: velocity, visibility, and versioning -- three virtues of successful ML deployments that MLEs learn to balance and grow as they mature. Finally, we discuss design implications and opportunities for future work.

arXiv admin note: text overlap with arXiv:2209.09125

References in corpus (8)

• Snorkel: Rapid Training Data Creation with Weak Supervision
• Improving fairness in machine learning systems: What do industry practitioners need?
• Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects
• Adoption and Effects of Software Engineering Best Practices in Machine Learning
• Data Vision: Learning to See Through Algorithmic Abstraction
• Machine Learning Operations (MLOps): Overview, Definition, and Architecture
• MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases
• Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development