Monitoring and explainability of models in production
arXiv:2007.06299
Abstract
The machine learning lifecycle extends beyond the deployment stage. Monitoring deployed models is crucial for continued provision of high quality machine learning enabled services. Key areas include model performance and data monitoring, detecting outliers and data drift using statistical techniques, and providing explanations of historic predictions. We discuss the challenges to successful implementation of solutions in each of these areas with some recent examples of production ready solutions using open source tools.
arXiv commentWorkshop on Challenges in Deploying and Monitoring Machine Learning Systems (ICML 2020)
References in corpus (10)
- Axiomatic Attribution for Deep Networks
- On Calibration of Modern Neural Networks
- Understanding Black-box Predictions via Influence Functions
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Exploring the Landscape of Spatial Robustness
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- Detecting and Correcting for Label Shift with Black Box Predictors
- Do Deep Generative Models Know What They Don't Know?
- Continual Learning in Practice
- Adversarial Detection and Correction by Matching Prediction Distributions