The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice

Abstract

Companies struggle to continuously develop and deploy AI models to complex production systems due to AI characteristics while assuring quality. To ease the development process, continuous pipelines for AI have become an active research area where consolidated and in-depth analysis regarding the terminology, triggers, tasks, and challenges is required. This paper includes a Multivocal Literature Review where we consolidated 151 relevant formal and informal sources. In addition, nine-semi structured interviews with participants from academia and industry verified and extended the obtained information. Based on these sources, this paper provides and compares terminologies for DevOps and CI/CD for AI, MLOps, (end-to-end) lifecycle management, and CD4ML. Furthermore, the paper provides an aggregated list of potential triggers for reiterating the pipeline, such as alert systems or schedules. In addition, this work uses a taxonomy creation strategy to present a consolidated pipeline comprising tasks regarding the continuous development of AI. This pipeline consists of four stages: Data Handling, Model Learning, Software Development and System Operations. Moreover, we map challenges regarding pipeline implementation, adaption, and usage for the continuous development of AI to these four stages.

accepted in the Journal Systems and Software

References in corpus (14)

• Robust Real-World Image Super-Resolution against Adversarial Attacks
• Machine Learning Operations (MLOps): Overview, Definition, and Architecture
• Software engineering for artificial intelligence and machine learning software: A systematic literature review
• A Data Quality-Driven View of MLOps
• MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases
• Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities
• CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking
• Towards Testing of Deep Learning Systems with Training Set Reduction
• Reliable Fleet Analytics for Edge IoT Solutions
• Quantitative Overfitting Management for Human-in-the-loop ML Application Development with ease.ml/meter
• ISTHMUS: Secure, Scalable, Real-time and Robust Machine Learning Platform for Healthcare
• Green Lighting ML: Confidentiality, Integrity, and Availability of Machine Learning Systems in Deployment
• Concept for a Technical Infrastructure for Management of Predictive Models in Industrial Applications
• Hidden Technical Debts for Fair Machine Learning in Financial Services