Technology Readiness Levels for Machine Learning Systems

Abstract

The development and deployment of machine learning (ML) systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. The lack of diligence can lead to technical debt, scope creep and misaligned objectives, model misuse and failures, and expensive consequences. Engineering systems, on the other hand, follow well-defined processes and testing standards to streamline development for high-quality, reliable results. The extreme is spacecraft systems, where mission critical measures and robustness are ingrained in the development process. Drawing on experience in both spacecraft engineering and ML (from research through product across domain areas), we have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" (MLTRL) framework defines a principled process to ensure robust, reliable, and responsible systems while being streamlined for ML workflows, including key distinctions from traditional software engineering. Even more, MLTRL defines a lingua franca for people across teams and organizations to work collaboratively on artificial intelligence and machine learning technologies. Here we describe the framework and elucidate it with several real world use-cases of developing ML methods from basic research through productization and deployment, in areas such as medical diagnostics, consumer computer vision, satellite imagery, and particle physics.

References in corpus (11)

• Event generation with SHERPA 1.1
• Model Cards for Model Reporting
• The frontier of simulation-based inference
• Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• A generic framework for privacy preserving deep learning
• Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
• Unity Perception: Generate Synthetic Data for Computer Vision
• Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale
• Bias and high-dimensional adjustment in observational studies of peer effects
• Learnings from Frontier Development Lab and SpaceML -- AI Accelerators for NASA and ESA

Cited by in corpus (8)

• The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice
• Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
• mlpack 4: a fast, header-only C++ machine learning library
• Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities
• Mapping the Landscape of Generative AI in Network Monitoring and Management
• Materiality and Risk in the Age of Pervasive AI Sensors
• Talking About the Assumption in the Room
• Reinforcement Learning-Based Production Scheduling in an Industry-Based Coating Scenario Using the Digital Model Playground