Underspecification Presents Challenges for Credibility in Modern Machine Learning

Abstract

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Updates: Updated statistical analysis in Section 6; Additional citations

References in corpus (3)

• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Capacity and Trainability in Recurrent Neural Networks

Cited by in corpus (9)

• Learning Transferable Visual Models From Natural Language Supervision
• Alignment of Language Agents
• Counterfactual Generative Networks
• Supervised Transfer Learning at Scale for Medical Imaging
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• The Privatization of AI Research(-ers): Causes and Potential Consequences -- From university-industry interaction to public research brain-drain?
• I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
• Optical Convolutional Neural Networks -- Combining Silicon Photonics and Fourier Optics for Computer Vision
• Neural Networks for Learning Counterfactual G-Invariances from Single Environments