All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously

Abstract

Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model $f(\mathbf{x})={\mathbf{x}}^T\beta$ with a fixed coefficient vector $\beta$) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.

The final, published article is now available at http://www.jmlr.org/papers/v20/18-760.html . This version contains minor changes to the introductory sections, and some typo corrections. The title of this article changed twice during the revision process (see v1 and v3 on arxiv)

References in corpus (5)

• Matching Methods for Causal Inference: A Review and a Look Forward
• Grouped variable importance with random forests and application to multiple functional data analysis
• Robust Optimization using Machine Learning for Uncertainty Sets
• A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results
• Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models

Cited by in corpus (96)

• A Survey on the Explainability of Supervised Machine Learning
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• Fairness in Machine Learning: A Survey
• Concept Whitening for Interpretable Image Recognition
• Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis
• Adversarial attacks and defenses in explainable artificial intelligence: A survey
• The impact of feature importance methods on the interpretation of defect classifiers
• FairSight: Visual Analytics for Fairness in Decision Making
• SurvSHAP(t): Time-dependent explanations of machine learning survival models
• Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process
• Model-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach
• Development and evaluation of an Explainable Prediction Model for Chronic Kidney Disease Patients based on Ensemble Trees
• Explainable Artificial Intelligence Approaches: A Survey
• On the Existence of Simpler Machine Learning Models
• Grouped Feature Importance and Combined Features Effect Plot
• Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretability
• Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?
• Decoding pedestrian and automated vehicle interactions using immersive virtual reality and interpretable deep learning
• survex: an R package for explaining machine learning survival models
• Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams
• New Paradigms for Exploiting Parallel Experiments in Bayesian Optimization
• The Grammar of Interactive Explanatory Model Analysis
• Interpretable Battery Cycle Life Range Prediction Using Early Degradation Data at Cell Level
• Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability
• Predicting Coronal Mass Ejections Using SDO/HMI Vector Magnetic Data Products and Recurrent Neural Networks
• Sampling, Intervention, Prediction, Aggregation: A Generalized Framework for Model-Agnostic Interpretations
• Predictive Multiplicity in Classification
• Considerations When Learning Additive Explanations for Black-Box Models
• A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study
• On Wasted Contributions: Understanding the Dynamics of Contributor-Abandoned Pull Requests
• Shapley Flow: A Graph-based Approach to Interpreting Model Predictions
• Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2
• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• Spatially Regularized Parametric Map Reconstruction for Fast Magnetic Resonance Fingerprinting
• Explaining Human Activity Recognition with SHAP: Validating Insights with Perturbation and Quantitative Measures
• Relative Feature Importance
• PS^2-Net: A Locally and Globally Aware Network for Point-Based Semantic Segmentation
• Constructing dynamic residential energy lifestyles using Latent Dirichlet Allocation
• Understanding transit ridership in an equity context through a comparison of statistical and machine learning algorithms
• TimberTrek: Exploring and Curating Sparse Decision Trees with Interactive Visualization
• Beyond development: Challenges in deploying machine learning models for structural engineering applications
• Equalizing Credit Opportunity in Algorithms: Aligning Algorithmic Fairness Research with U.S. Fair Lending Regulation
• SLISEMAP: Supervised dimensionality reduction through local explanations
• Feature Inference Attack on Shapley Values
• Exploring the Whole Rashomon Set of Sparse Decision Trees
• Rational Shapley Values
• Evaluating the Correctness of Explainable AI Algorithms for Classification
• On the computation of counterfactual explanations -- A survey
• Interpretable machine learning for time-to-event prediction in medicine and healthcare
• Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints
• Tagging more quark jet flavours at FCC-ee at 91 GeV with a transformer-based neural network
• Hybrid analytic and machine-learned baryonic property insertion into galactic dark matter haloes
• An Artificial Intelligence-based model for cell killing prediction: development, validation and explainability analysis of the ANAKIN model
• Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models
• Key predictors for climate policy support and political mobilization: The role of beliefs and preferences
• Graph Isomorphic Networks for Assessing Reliability of the Medium-Voltage Grid
• Characterizing Fairness Over the Set of Good Models Under Selective Labels
• Loss-Aversively Fair Classification
• Accounting for Model Uncertainty in Algorithmic Discrimination
• AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Models
• Explaining deep neural network models for electricity price forecasting with XAI
• Efficient computation of counterfactual explanations of LVQ models
• LioNets: A Neural-Specific Local Interpretation Technique Exploiting Penultimate Layer Information
• Relaxed Parameter Sharing: Effectively Modeling Time-Varying Relationships in Clinical Time-Series
• High-dimensional reinforcement learning for optimization and control of ultracold quantum gases
• Prediction of soft proton intensities in the near-Earth space using machine learning
• Towards Explainability of Machine Learning Models in Insurance Pricing
• Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data
• Machine Learning, Deep Learning, and Hedonic Methods for Real Estate Price Prediction
• MeLIME: Meaningful Local Explanation for Machine Learning Models
• Forks Over Knives: Predictive Inconsistency in Criminal Justice Algorithmic Risk Assessment Tools
• Randomized Ablation Feature Importance
• Surface solar radiation: AI satellite retrieval can outperform Heliosat and generalizes well to other climate zones
• An Explainable Multi-Task Similarity Measure: Integrating Accumulated Local Effects and Weighted Fréchet Distance
• Performance is not enough: the story told by a Rashomon quartet
• Interpreting Vision and Language Generative Models with Semantic Visual Priors
• The Intriguing Relation Between Counterfactual Explanations and Adversarial Examples
• Assessing variable importance in survival analysis using machine learning
• LionForests: Local Interpretation of Random Forests
• AI-Spectra: A Visual Dashboard for Model Multiplicity to Enhance Informed and Transparent Decision-Making
• Cross-model Fairness: Empirical Study of Fairness and Ethics Under Model Multiplicity
• Assessing Workers Neuro-physiological Stress Responses to Augmented Reality Safety Warnings in Immersive Virtual Roadway Work Zones
• Impact of Legal Requirements on Explainability in Machine Learning
• Structural importance and evolution: an application to financial transaction networks
• Scalable Rule Lists Learning with Sampling
• Conditional independence testing: a predictive perspective
• Factor Importance Ranking and Selection using Total Indices
• Uncertainty-aware Sensitivity Analysis Using Rényi Divergences
• Explainable AI for Classification using Probabilistic Logic Inference
• Hierarchical causal variance decomposition for institution and provider comparisons in healthcare
• New Perspective of Interpretability of Deep Neural Networks
• High-Dimensional Inference Based on the Leave-One-Covariate-Out LASSO Path
• Personalised dynamic super learning: an application in predicting hemodiafiltration convection volumes
• Interpretable Artificial Intelligence through the Lens of Feature Interaction
• "A 6 or a 9?": Ensemble Learning Through the Multiplicity of Performant Models and Explanations