Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey

Abstract

Using deep learning models to diagnose cancer from histology data presents several challenges. Cancer grading and localization of regions of interest (ROIs) in these images normally relies on both image- and pixel-level labels, the latter requiring a costly annotation process. Deep weakly-supervised object localization (WSOL) methods provide different strategies for low-cost training of deep learning models. Using only image-class annotations, these methods can be trained to classify an image, and yield class activation maps (CAMs) for ROI localization. This paper provides a review of state-of-art DL methods for WSOL. We propose a taxonomy where these methods are divided into bottom-up and top-down methods according to the information flow in models. Although the latter have seen limited progress, recent bottom-up methods are currently driving much progress with deep WSOL methods. Early works focused on designing different spatial pooling functions. However, these methods reached limited localization accuracy, and unveiled a major limitation -- the under-activation of CAMs which leads to high false negative localization. Subsequent works aimed to alleviate this issue and recover complete object. Representative methods from our taxonomy are evaluated and compared in terms of classification and localization accuracy on two challenging histology datasets. Overall, the results indicate poor localization performance, particularly for generic methods that were initially designed to process natural images. Methods designed to address the challenges of histology data yielded good results. However, all methods suffer from high false positive/negative localization. Four key challenges are identified for the application of deep WSOL methods in histology -- under/over activation of CAMs, sensitivity to thresholding, and model selection.

arXiv commentAccepted for publication at the Journal of Machine Learning for Biomedical Imaging (MELBA) https://melba-journal.org/2023:004

References in corpus (69)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• A Survey on Deep Learning in Medical Image Analysis
• Neural Machine Translation by Jointly Learning to Align and Translate
• mixup: Beyond Empirical Risk Minimization
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Improved Regularization of Convolutional Neural Networks with Cutout
• Interpretable machine learning: definitions, methods, and applications
• Understanding deep learning requires rethinking generalization
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• Machine learning methods for histopathological image analysis
• Deep neural network models for computational histopathology: A survey
• Multiple Instance Learning: A Survey of Problem Characteristics and Applications
• BACH: Grand Challenge on Breast Cancer Histology Images
• Attention-based Deep Multiple Instance Learning
• High-Resolution Representations for Labeling Pixels and Regions
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Sanity Checks for Saliency Maps
• Deep Label Distribution Learning with Label Ambiguity
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Training Convolutional Networks with Noisy Labels
• Revisiting Multiple Instance Neural Networks
• Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
• 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study
• Deep Learning for Whole Slide Image Analysis: An Overview
• Constrained-CNN losses for weakly supervised segmentation
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• A Deep Learning Interpretable Classifier for Diabetic Retinopathy Disease Grading
• Resolving challenges in deep learning-based analyses of histopathological images using explanation methods
• The (Un)reliability of saliency methods
• Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodules
• Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models
• Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs
• DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation
• Classification and Disease Localization in Histopathology Using Only Global Labels: A Weakly-Supervised Approach
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization
• Learning from Noisy Labels with Deep Neural Networks: A Survey
• Real Time Image Saliency for Black Box Classifiers
• Cancer Metastasis Detection With Neural Conditional Random Field
• Adversarial Complementary Learning for Weakly Supervised Object Localization
• Curriculum semi-supervised segmentation
• Tell Me Where to Look: Guided Attention Inference Network
• Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup
• Normalized Cut Loss for Weakly-supervised CNN Segmentation
• SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization
• What's the Point: Semantic Segmentation with Point Supervision
• Two-Phase Learning for Weakly Supervised Object Localization
• Weakly Supervised Instance Segmentation using Class Peak Response
• A Survey on Neural Network Interpretability
• IS-CAM: Integrated Score-CAM for axiomatic-based explanations
• Understanding Integrated Gradients with SmoothTaylor for Deep Neural Network Attribution
• Learning Functional Causal Models with Generative Neural Networks
• Soft Proposal Networks for Weakly Supervised Object Localization
• Convolutional Neural Networks for Histopathology Image Classification: Training vs. Using Pre-Trained Networks
• Evaluating Weakly Supervised Object Localization Methods Right
• Inter-Image Communication for Weakly Supervised Localization
• Shallow Feature Matters for Weakly Supervised Object Localization
• Automatic labeling of molecular biomarkers of whole slide immunohistochemistry images using fully convolutional networks
• Min-max Entropy for Weakly Supervised Pointwise Localization
• Geometry Constrained Weakly Supervised Object Localization
• ProNet: Learning to Propose Object-specific Boxes for Cascaded Neural Networks
• Rethinking the Route Towards Weakly Supervised Object Localization
• Negative Evidence Matters in Interpretable Histology Image Classification
• Combinational Class Activation Maps for Weakly Supervised Object Localization
• Constrained Sampling for Class-Agnostic Weakly Supervised Object Localization
• Discriminative Sampling of Proposals in Self-Supervised Transformers for Weakly Supervised Object Localization

Cited by in corpus (5)

• Deep neural network models for computational histopathology: A survey
• Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification
• Weakly Supervised Attention-based Models Using Activation Maps for Citrus Mite and Insect Pest Classification
• DiPS: Discriminative Pseudo-Label Sampling with Self-Supervised Transformers for Weakly Supervised Object Localization
• A Realistic Protocol for Evaluation of Weakly Supervised Object Localization