A systematic study of the class imbalance problem in convolutional neural networks

Abstract

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Striving for Simplicity: The All Convolutional Net
• ImageNet pre-trained models with batch normalization
• The iNaturalist Species Classification and Detection Dataset

Cited by in corpus (229)

• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Deep Learning Algorithms for Rotating Machinery Intelligent Diagnosis: An Open Source Benchmark Study
• Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks
• Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm
• Detecting Mammals in UAV Images: Best Practices to address a substantially Imbalanced Dataset with Deep Learning
• Balanced Meta-Softmax for Long-Tailed Visual Recognition
• Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions
• Analyzing Overfitting under Class Imbalance in Neural Networks for Image Segmentation
• SCADA System Testbed for Cybersecurity Research Using Machine Learning Approach
• Detecting and interpreting myocardial infarction using fully convolutional neural networks
• A Survey on Long-Tailed Visual Recognition
• FETCH: A deep-learning based classifier for fast transient classification
• DR$|$GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images
• Class-Balanced Loss Based on Effective Number of Samples
• Delving into Deep Imbalanced Regression
• Assessment of Deep Convolutional Neural Networks for Road Surface Classification
• Deep learning predictions of galaxy merger stage and the importance of observational realism
• Deep Multi-Magnification Networks for Multi-Class Breast Cancer Image Segmentation
• PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation
• Deep learning guided image-based droplet sorting for on-demand selection and analysis of single cells and 3D cell cultures
• Detection of masses and architectural distortions in digital breast tomosynthesis: a publicly available dataset of 5,060 patients and a deep learning model
• Toward Open-World Electroencephalogram Decoding Via Deep Learning: A Comprehensive Survey
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• Personalized Dialogue Generation with Diversified Traits
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Seizure Detection using Least EEG Channels by Deep Convolutional Neural Network
• BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition
• Deep convolutional neural networks for multi-scale time-series classification and application to disruption prediction in fusion devices
• Differential Privacy Has Disparate Impact on Model Accuracy
• Active, Continual Fine Tuning of Convolutional Neural Networks for Reducing Annotation Efforts
• Self-supervised Learning is More Robust to Dataset Imbalance
• Augmentation for small object detection
• Learning to Amend Facial Expression Representation via De-albino and Affinity
• Convolutional Neural Networks based Focal Loss for Class Imbalance Problem: A Case Study of Canine Red Blood Cells Morphology Classification
• Data Augmentation for Object Detection via Progressive and Selective Instance-Switching
• Resolving Class Imbalance in Object Detection with Weighted Cross Entropy Losses
• Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning
• Long-tail learning via logit adjustment
• Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization
• Topology-Imbalance Learning for Semi-Supervised Node Classification
• ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning
• Curiosity-Driven Experience Prioritization via Density Estimation
• The PAU Survey: Photometric redshifts using transfer learning from simulations
• FINE Samples for Learning with Noisy Labels
• GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging
• Application of Decision Rules for Handling Class Imbalance in Semantic Segmentation
• Class-Imbalanced Semi-Supervised Learning
• Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspective
• Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets
• CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning
• Analyzing the Usefulness of the DARPA OpTC Dataset in Cyber Threat Detection Research
• Maintaining Discrimination and Fairness in Class Incremental Learning
• Rethinking of Pedestrian Attribute Recognition: Realistic Datasets with Efficient Method
• Boosting-GNN: Boosting Algorithm for Graph Networks on Imbalanced Node Classification
• Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention
• Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning
• Pulsar Candidates Classification with Deep Convolutional Neural Networks
• On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
• Dual-Sampling Attention Network for Diagnosis of COVID-19 from Community Acquired Pneumonia
• DBN-Mix: Training Dual Branch Network Using Bilateral Mixup Augmentation for Long-Tailed Visual Recognition
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition
• Learning Imbalanced Datasets with Maximum Margin Loss
• Disentangling Label Distribution for Long-tailed Visual Recognition
• Cost-Sensitive Convolution based Neural Networks for Imbalanced Time-Series Classification
• ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot Learning
• Data Augmentation and Classification of Sea-Land Clutter for Over-the-Horizon Radar Using AC-VAEGAN
• Deep Learning for Scene Classification: A Survey
• Clustering and Learning from Imbalanced Data
• Jensen-Shannon Divergence Based Novel Loss Functions for Bayesian Neural Networks
• Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
• Argument Component Classification for Classroom Discussions
• A Unified Deep Learning Framework for Short-Duration Speaker Verification in Adverse Environments
• Automatically Detecting Self-Reported Birth Defect Outcomes on Twitter for Large-scale Epidemiological Research
• Class-Imbalanced Complementary-Label Learning via Weighted Loss
• Robustness May Be at Odds with Fairness: An Empirical Study on Class-wise Accuracy
• Cross-hospital Sepsis Early Detection via Semi-supervised Optimal Transport with Self-paced Ensemble
• A dual-branch model with inter- and intra-branch contrastive loss for long-tailed recognition
• Multi-task Regularization Based on Infrequent Classes for Audio Captioning
• Connecting optical morphology, environment, and HI mass fraction for low-redshift galaxies using deep learning
• Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification
• MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition
• ELF: An Early-Exiting Framework for Long-Tailed Classification
• The Devil is the Classifier: Investigating Long Tail Relation Classification with Decoupling Analysis
• Adaptive Learning for the Resource-Constrained Classification Problem
• Targeted Supervised Contrastive Learning for Long-Tailed Recognition
• Enhancing Robustness of Deep Neural Networks Against Adversarial Malware Samples: Principles, Framework, and AICS'2019 Challenge
• Cell nuclei classification in histopathological images using hybrid OLConvNet
• Euclid Preparation XXXIII. Characterization of convolutional neural networks for the identification of galaxy-galaxy strong lensing events
• Transfer Learning for Oral Cancer Detection using Microscopic Images
• Minority Class Oriented Active Learning for Imbalanced Datasets
• Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
• Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
• Virtual Classification: Modulating Domain-Specific Knowledge for Multidomain Crowd Counting
• Feature Space Augmentation for Long-Tailed Data
• Automatic Diagnosis of Pneumothorax from Chest Radiographs: A Systematic Literature Review
• Bounding Boxes Are All We Need: Street View Image Classification via Context Encoding of Detected Buildings
• Analyzing Differentiable Fuzzy Logic Operators
• CNN-based event classification for alpha-decay events in nuclear emulsion
• Deep Forest: Neural Network reconstruction of the Lyman-alpha forest
• Rethinking Importance Weighting for Deep Learning under Distribution Shift
• CardiSort: a convolutional neural network for cross vendor automated sorting of cardiac MR images
• Self-Supervised Features Improve Open-World Learning
• Analysis of Vision-based Abnormal Red Blood Cell Classification
• Balanced Knowledge Distillation for Long-tailed Learning
• Galaxy Image Classification using Hierarchical Data Learning with Weighted Sampling and Label Smoothing
• ReMix: Calibrated Resampling for Class Imbalance in Deep learning
• The Deep Arbitrary Polynomial Chaos Neural Network or how Deep Artificial Neural Networks could benefit from Data-Driven Homogeneous Chaos Theory
• Imbalanced Continual Learning with Partitioning Reservoir Sampling
• Improving Calibration for Long-Tailed Recognition
• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• Balanced-MixUp for Highly Imbalanced Medical Image Classification
• Adversarial Robustness under Long-Tailed Distribution
• LapTool-Net: A Contextual Detector of Surgical Tools in Laparoscopic Videos Based on Recurrent Convolutional Neural Networks
• Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
• Long-tailed Distribution Adaptation
• Enhanced prediction of spine surgery outcomes using advanced machine learning techniques and oversampling methods
• Teaching Robots to Build Simulations of Themselves
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data
• U-Net for MAV-based Penstock Inspection: an Investigation of Focal Loss in Multi-class Segmentation for Corrosion Identification
• A Failure of Aspect Sentiment Classifiers and an Adaptive Re-weighting Solution
• MetaFusion: Controlled False-Negative Reduction of Minority Classes in Semantic Segmentation
• A Comparison of Synthetic Oversampling Methods for Multi-class Text Classification
• Machine Learning Applications to Kronian Magnetospheric Reconnection Classification
• Meta Feature Modulator for Long-tailed Recognition
• Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data
• Prototypical Clustering Networks for Dermatological Disease Diagnosis
• End-to-End ASR for Code-switched Hindi-English Speech
• Cost-Sensitive BERT for Generalisable Sentence Classification with Imbalanced Data
• Composer Style Classification of Piano Sheet Music Images Using Language Model Pretraining
• A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning
• FLEX: Unifying Evaluation for Few-Shot NLP
• Towards Resolving the Challenge of Long-tail Distribution in UAV Images for Object Detection
• Teacher's pet: understanding and mitigating biases in distillation
• DT-DDNN: A Physical Layer Security Attack Detector in 5G RF Domain for CAVs
• Towards Overcoming False Positives in Visual Relationship Detection
• Robustness in Compressed Neural Networks for Object Detection
• Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
• Addressing Class Imbalance in Federated Learning
• Label-Aware Distribution Calibration for Long-tailed Classification
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Cosmic Ray Rejection with Attention Augmented Deep Learning
• LLMs for Drug-Drug Interaction Prediction: A Comprehensive Comparison
• An Empirical Study of the Imbalance Issue in Software Vulnerability Detection
• An Automatic Text Classification Method Based on Hierarchical Taxonomies, Neural Networks and Document Embedding: The NETHIC Tool
• Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Rates
• The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation
• Deep Learning-Based Strategy for Macromolecules Classification with Imbalanced Data from Cellular Electron Cryotomography
• Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier
• Mixup Regularization for Region Proposal based Object Detectors
• A surrogate loss function for optimization of $F_{\beta }$ score in binary classification with imbalanced data
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• Natural vs Balanced Distribution in Deep Learning on Whole Slide Images for Cancer Detection
• Distributional Robustness Loss for Long-tail Learning
• Does Head Label Help for Long-Tailed Multi-Label Text Classification
• Majority or Minority: Data Imbalance Learning Method for Named Entity Recognition
• A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization
• Sequential Targeting: an incremental learning approach for data imbalance in text classification
• Investigate the Essence of Long-Tailed Recognition from a Unified Perspective
• Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
• Automated Respiratory Event Detection Using Deep Neural Networks
• Posterior Re-calibration for Imbalanced Datasets
• Text Data Augmentation: Towards better detection of spear-phishing emails
• Precisely Predicting Acute Kidney Injury with Convolutional Neural Network Based on Electronic Health Record Data
• Adjusting Decision Boundary for Class Imbalanced Learning
• Neural Style Transfer for Point Clouds
• SNU_IDS at SemEval-2019 Task 3: Addressing Training-Test Class Distribution Mismatch in Conversational Classification
• Cavity Filling: Pseudo-Feature Generation for Multi-Class Imbalanced Data Problems in Deep Learning
• SelectNet: Learning to Sample from the Wild for Imbalanced Data Training
• From Generalized zero-shot learning to long-tail with class descriptors
• Interpretable and Flexible Target-Conditioned Neural Planners For Autonomous Vehicles
• Switching Loss for Generalized Nucleus Detection in Histopathology
• Centralized Copy-Paste: Enhanced Data Augmentation Strategy for Wildland Fire Semantic Segmentation
• Low-Resource Text-to-Speech Synthesis Using Noise-Augmented Training of ForwardTacotron
• MosaicOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection
• Continuous Weight Balancing
• Trustworthy Long-Tailed Classification
• Using Artificial Intelligence to Recapture Norms: Did #metoo change gender norms in Sweden?
• Convolutional Dictionary Pair Learning Network for Image Representation Learning
• Analysis of skin lesion images with deep learning
• F^2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax
• Transfer Learning for Aided Target Recognition: Comparing Deep Learning to other Machine Learning Approaches
• Multi-Agent Semi-Siamese Training for Long-tail and Shallow Face Learning
• Explanation and Use of Uncertainty Quantified by Bayesian Neural Network Classifiers for Breast Histopathology Images
• VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition
• Improving traffic sign recognition by active search
• Subtask-dominated Transfer Learning for Long-tail Person Search
• Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition
• Image anomaly detection with capsule networks and imbalanced datasets
• Addressing Weak Decision Boundaries in Image Classification by Leveraging Web Search and Generative Models
• Adaptive Class Weight based Dual Focal Loss for Improved Semantic Segmentation
• Class-dependent Compression of Deep Neural Networks
• Investigating Group Distributionally Robust Optimization for Deep Imbalanced Learning: A Case Study of Binary Tabular Data Classification
• DropClass and DropAdapt: Dropping classes for deep speaker representation learning
• Parametric Contrastive Learning
• Energy Aligning for Biased Models
• PML: Progressive Margin Loss for Long-tailed Age Classification
• Photometric Redshift Estimation with Convolutional Neural Networks and Galaxy Images: A Case Study of Resolving Biases in Data-Driven Methods
• Domain Adaptation for Rare Classes Augmented with Synthetic Samples
• Addressing Class Imbalance in Scene Graph Parsing by Learning to Contrast and Score
• Self Supervision to Distillation for Long-Tailed Visual Recognition
• Deep Adversarial Domain Adaptation Based on Multi-layer Joint Kernelized Distance
• Imbalanced data preprocessing techniques utilizing local data characteristics
• Adaptive Cost-Sensitive Learning in Neural Networks for Misclassification Cost Problems
• Spatiotemporal Ground Reaction Force Analysis using Convolutional Neural Networks to Analyze Parkinsonian Gait
• Deep ensemble network with explicit complementary model for accuracy-balanced classification
• CRCEN: A Generalized Cost-sensitive Neural Network Approach for Imbalanced Classification
• Time Series Classification Using Convolutional Neural Network On Imbalanced Datasets
• Training Over-parameterized Models with Non-decomposable Objectives
• A survey on deep learning approaches for breast cancer diagnosis
• Influence-Balanced Loss for Imbalanced Visual Classification
• How saccadic vision might help with theinterpretability of deep networks
• How Sensitive are Meta-Learners to Dataset Imbalance?
• SyReNets: Symbolic Residual Neural Networks
• Procrustean Training for Imbalanced Deep Learning
• Boost Neural Networks by Checkpoints
• RSG: A Simple but Effective Module for Learning Imbalanced Datasets
• Offline Clustering Approach to Self-supervised Learning for Class-imbalanced Image Data
• Slice Tuner: A Selective Data Acquisition Framework for Accurate and Fair Machine Learning Models
• Classification of optics-free images with deep neural networks
• Mitigating Data Imbalance in Automated Speaking Assessment
• BiSTF: Bilateral-Branch Self-Training Framework for Semi-Supervised Large-scale Fine-Grained Recognition
• Comparative Analysis of Deep Learning Architectures for Multi-Disease Classification of Single-Label Chest X-rays
• MV-C3D: A Spatial Correlated Multi-View 3D Convolutional Neural Networks
• Unifying data for fine-grained visual species classification