Model Patching: Closing the Subgroup Performance Gap with Data Augmentation

Abstract

Classifiers in machine learning are often brittle when deployed. Particularly concerning are models with inconsistent performance on specific subgroups of a class, e.g., exhibiting disparities in skin cancer classification in the presence or absence of a spurious bandage. To mitigate these performance differences, we introduce model patching, a two-stage framework for improving robustness that encourages the model to be invariant to subgroup differences, and focus on class information shared by subgroups. Model patching first models subgroup features within a class and learns semantic transformations between them, and then trains a classifier with data augmentations that deliberately manipulate subgroup features. We instantiate model patching with CAMEL, which (1) uses a CycleGAN to learn the intra-class, inter-subgroup augmentations, and (2) balances subgroup performance using a theoretically-motivated subgroup consistency regularizer, accompanied by a new robust objective. We demonstrate CAMEL's effectiveness on 3 benchmark datasets, with reductions in robust error of up to 33% relative to the best baseline. Lastly, CAMEL successfully patches a model that fails due to spurious features on a real-world skin cancer dataset.

References in corpus (19)

• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Unsupervised Data Augmentation for Consistency Training
• Conditional Adversarial Domain Adaptation
• Data Augmentation Generative Adversarial Networks
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Invariant Risk Minimization
• BAGAN: Data Augmentation with Balancing GAN
• Adversarial Transformation Networks: Learning to Generate Adversarial Examples
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• A Kernel Theory of Modern Data Augmentation
• Learning Data Manipulation for Augmentation and Weighting
• Deep Manifold Traversal: Changing Labels with Convolutional Features
• Adversarial Learning of General Transformations for Data Augmentation
• Generative Models For Deep Learning with Very Scarce Data

Cited by in corpus (12)

• On the Opportunities and Risks of Foundation Models
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Representation Bias in Data: A Survey on Identification and Resolution Techniques
• Learning Domain Invariant Representations by Joint Wasserstein Distance Minimization
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• A Fine-Grained Analysis on Distribution Shift
• Robustness through Data Augmentation Loss Consistency
• Combining Data-driven Supervision with Human-in-the-loop Feedback for Entity Resolution
• Focus on the Common Good: Group Distributional Robustness Follows
• Finding and Fixing Spurious Patterns with Explanations
• Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems
• Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment