Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Abstract

With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to assess, visualize, and improve UQ. Uncertainty Toolbox additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references. We hope that this toolbox is useful for accelerating and uniting research efforts in uncertainty in machine learning.

References in corpus (8)

• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Evaluating epidemic forecasts in an interval format
• MOPO: Model-based Offline Policy Optimization
• Single-Model Uncertainties for Deep Learning
• Calibration for the (Computationally-Identifiable) Masses
• Calibrated Model-Based Deep Reinforcement Learning
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Reliable training and estimation of variance networks

Cited by in corpus (6)

• Uncertainty Quantification for Molecular Property Predictions with Graph Neural Architecture Search
• High-dimensional Bayesian Optimization of Hyperparameters for an Attention-based Network to Predict Materials Property: a Case Study on CrabNet using Ax and SAASBO
• Prediction uncertainty validation for computational chemists
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• A Hybrid Probabilistic Battery Health Management Approach for Robust Inspection Drone Operations
• On the role of Model Uncertainties in Bayesian Optimization