Underspecification Presents Challenges for Credibility in Modern Machine Learning

Abstract

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Updates: Updated statistical analysis in Section 6; Additional citations

References in corpus (20)

• Shortcut Learning in Deep Neural Networks
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Counterfactual Fairness
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy
• Deep Ensembles: A Loss Landscape Perspective
• Invariant Risk Minimization
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Natural Adversarial Examples
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Causality for Machine Learning
• Measuring and Reducing Gendered Correlations in Pre-trained Models
• The Consciousness Prior
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Capacity and Trainability in Recurrent Neural Networks
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Predictive Multiplicity in Classification

Cited by in corpus (74)

• Learning Transferable Visual Models From Natural Language Supervision
• Metrics reloaded: Recommendations for image analysis validation
• Dawn of the transformer era in speech emotion recognition: closing the valence gap
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Technology Readiness Levels for Machine Learning Systems
• Bridging observation, theory and numerical simulation of the ocean using Machine Learning
• A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
• Harms from Increasingly Agentic Algorithmic Systems
• Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing
• On the Existence of Simpler Machine Learning Models
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Physics-informed machine learning for building performance simulation-A review of a nascent field
• Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Alignment of Language Agents
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Counterfactual Generative Networks
• Lifelong Self-Adaptation: Self-Adaptation Meets Lifelong Machine Learning
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• Supervised Transfer Learning at Scale for Medical Imaging
• Beyond development: Challenges in deploying machine learning models for structural engineering applications
• Deep Learning Body Region Classification of MRI and CT examinations
• Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools
• The MultiBERTs: BERT Reproductions for Robustness Analysis
• Desiderata for Representation Learning: A Causal Perspective
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• Learning Neural Network Subspaces
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• Fooling Partial Dependence via Data Poisoning
• A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness
• Overparameterized Linear Regression under Adversarial Attacks
• In Search of netUnicorn: A Data-Collection Platform to Develop Generalizable ML Models for Network Security Problems
• Undesirable Biases in NLP: Addressing Challenges of Measurement
• Arbitrary Decisions are a Hidden Cost of Differentially Private Training
• AI and Ethics -- Operationalising Responsible AI
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Medical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Is Machine Learning Unsafe and Irresponsible in Social Sciences? Paradoxes and Reconsidering from Recidivism Prediction Tasks
• Beyond Fairness Metrics: Roadblocks and Challenges for Ethical AI in Practice
• Artificial Intelligence in Healthcare: Lost In Translation?
• Consensus statement on the credibility assessment of ML predictors
• Robustness testing of AI systems: A case study for traffic sign recognition
• Consistent Counterfactuals for Deep Models
• Understanding Prediction Discrepancies in Machine Learning Classifiers
• Improving Fairness in Criminal Justice Algorithmic Risk Assessments Using Conformal Prediction Sets
• Interpretable and synergistic deep learning for visual explanation and statistical estimations of segmentation of disease features from medical images
• Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding
• Grounding Representation Similarity with Statistical Testing
• What Constitutes a Less Discriminatory Algorithm?
• Distinguishing rule- and exemplar-based generalization in learning systems
• Controlling Neural Networks with Rule Representations
• A neural anisotropic view of underspecification in deep learning
• The Privatization of AI Research(-ers): Causes and Potential Consequences -- From university-industry interaction to public research brain-drain?
• Experimental Standards for Deep Learning in Natural Language Processing Research
• Selective Ensembles for Consistent Predictions
• Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding
• Can you tell? SSNet -- a Sagittal Stratum-inspired Neural Network Framework for Sentiment Analysis
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Optical Convolutional Neural Networks -- Combining Silicon Photonics and Fourier Optics for Computer Vision
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Neural Networks for Learning Counterfactual G-Invariances from Single Environments
• Blood Glucose Control Via Pre-trained Counterfactual Invertible Neural Networks
• Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
• Extracting Global Dynamics of Loss Landscape in Deep Learning Models
• A Topological-Framework to Improve Analysis of Machine Learning Model Performance
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Characterizing the risk of fairwashing
• Causal Models in Requirement Specifications for Machine Learning: A vision
• Covariate Shift in High-Dimensional Random Feature Regression
• Neural forecasting at scale