Deep neural network models for computational histopathology: A survey

Abstract

Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to diseases progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting cancer histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis. From the survey of over 130 papers, we review the fields progress based on the methodological aspect of different machine learning strategies such as supervised, weakly supervised, unsupervised, transfer learning and various other sub-variants of these methods. We also provide an overview of deep learning based survival models that are applicable for disease-specific prognosis tasks. Finally, we summarize several existing open datasets and highlight critical challenges and limitations with current deep learning approaches, along with possible avenues for future research.

arXiv commentPublished in Medical Image Analysis, Vol. 67, Jan 2021. (10.1016/j.media.2020.101813)

References in corpus (69)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Survey on Deep Learning in Medical Image Analysis
• Auto-Encoding Variational Bayes
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Fully Convolutional Networks for Semantic Segmentation
• Conditional Random Fields as Recurrent Neural Networks
• Deep Visual Domain Adaptation: A Survey
• Squeeze-and-Excitation Networks
• DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Generative Adversarial Network in Medical Imaging: A Review
• Unsupervised Representation Learning by Predicting Image Rotations
• Domain-adversarial neural networks to address the appearance variability of histopathology images
• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• Recurrent Models of Visual Attention
• Deep Learning for Identifying Metastatic Breast Cancer
• Automated Gleason Grading of Prostate Biopsies using Deep Learning
• BACH: Grand Challenge on Breast Cancer Histology Images
• Attention-based Deep Multiple Instance Learning
• What do we need to build explainable AI systems for the medical domain?
• Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology
• This Looks Like That: Deep Learning for Interpretable Image Recognition
• Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancer
• Robust Physical-World Attacks on Deep Learning Models
• Assessment of algorithms for mitosis detection in breast cancer histopathology images
• PhaseStain: Digital staining of label-free quantitative phase microscopy images using deep learning
• Action Recognition using Visual Attention
• MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images
• Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge
• Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks
• Neural Image Compression for Gigapixel Histopathology Image Analysis
• Constrained Deep Weak Supervision for Histopathology Image Segmentation
• Gland Instance Segmentation Using Deep Multichannel Neural Networks
• RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification
• Her2 Challenge Contest: A Detailed Assessment of Automated Her2 Scoring Algorithms in Whole Slide Images of Breast Cancer Tissues
• Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard
• Fully Convolutional Attention Networks for Fine-Grained Recognition
• Deep Multi-Magnification Networks for Multi-Class Breast Cancer Image Segmentation
• Unsupervised Learning for Cell-level Visual Representation in Histopathology Images with Generative Adversarial Networks
• Learning Where to See: A Novel Attention Model for Automated Immunohistochemical Scoring
• Cancer Metastasis Detection With Neural Conditional Random Field
• Neural Stain-Style Transfer Learning using GAN for Histopathological Images
• Dataset of Segmented Nuclei in Hematoxylin and Eosin Stained Histopathology Images of 10 Cancer Types
• An End-to-End Deep Learning Histochemical Scoring System for Breast Cancer Tissue Microarray
• Leveraging Uncertainty Estimates for Predicting Segmentation Quality
• A Closer Look at Domain Shift for Deep Learning in Histopathology
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• Terabyte-scale Deep Multiple Instance Learning for Classification and Localization in Pathology
• On Learning Where To Look
• An attention-based multi-resolution model for prostate whole slide imageclassification and localization
• BoxNet: Deep Learning Based Biomedical Image Segmentation Using Boxes Only Annotation
• Computer-aided diagnosis of lung carcinoma using deep learning - a pilot study
• MITOS-RCNN: A Novel Approach to Mitotic Figure Detection in Breast Cancer Histopathology Images using Region Based Convolutional Neural Networks
• CELNet: Evidence Localization for Pathology Images using Weakly Supervised Learning
• Unsupervised Prostate Cancer Detection on H&E using Convolutional Adversarial Autoencoders
• DASGAN -- Joint Domain Adaptation and Segmentation for the Analysis of Epithelial Regions in Histopathology PD-L1 Images
• Representation-Aggregation Networks for Segmentation of Multi-Gigapixel Histology Images
• Neural Ordinary Differential Equations for Semantic Segmentation of Individual Colon Glands
• Multimodal Multitask Representation Learning for Pathology Biobank Metadata Prediction
• Domain Adaptation-based Augmentation for Weakly Supervised Nuclei Detection
• Cluster-Based Learning from Weakly Labeled Bags in Digital Pathology
• Deep Learning Under the Microscope: Improving the Interpretability of Medical Imaging Neural Networks
• PFA-ScanNet: Pyramidal Feature Aggregation with Synergistic Learning for Breast Cancer Metastasis Analysis
• Fooling the Crowd with Deep Learning-based Methods
• Multi-Resolution Networks for Semantic Segmentation in Whole Slide Images

Cited by in corpus (34)

• Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy
• GasHis-Transformer: A Multi-scale Visual Transformer Approach for Gastric Histopathological Image Detection
• A Survey on Graph-Based Deep Learning for Computational Histopathology
• Self-supervised driven consistency training for annotation efficient histopathology image analysis
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction
• Gastric histopathology image segmentation using a hierarchical conditional random field
• Digitization of Pathology Labs: A Review of Lessons Learned
• RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• Objective Diagnosis for Histopathological Images Based on Machine Learning Techniques: Classical Approaches and New Trends
• Learning image representations for anomaly detection: application to discovery of histological alterations in drug development
• Learning to segment images with classification labels
• Learning to Downsample for Segmentation of Ultra-High Resolution Images
• Cyclic Learning: Bridging Image-level Labels and Nuclei Instance Segmentation
• Evaluating histopathology transfer learning with ChampKit
• Agent Aggregator with Mask Denoise Mechanism for Histopathology Whole Slide Image Analysis
• A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models
• GenSelfDiff-HIS: Generative Self-Supervision Using Diffusion for Histopathological Image Segmentation
• Towards Measuring Domain Shift in Histopathological Stain Translation in an Unsupervised Manner
• Preserving Dense Features for Ki67 Nuclei Detection
• Self-Contrastive Weakly Supervised Learning Framework for Prognostic Prediction Using Whole Slide Images
• Recognizing Magnification Levels in Microscopic Snapshots
• A Petri Dish for Histopathology Image Analysis
• An Attention-based Weakly Supervised framework for Spitzoid Melanocytic Lesion Diagnosis in WSI
• Foveation for Segmentation of Ultra-High Resolution Images
• Morphomics via Next-generation Electron Microscopy
• An Accelerated Pipeline for Multi-label Renal Pathology Image Segmentation at the Whole Slide Image Level
• Resource and data efficient self supervised learning
• Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imaging
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• MMAP: A Multi-Magnification and Prototype-Aware Architecture for Predicting Spatial Gene Expression
• SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classification
• Conditional Deep Convolutional Neural Networks for Improving the Automated Screening of Histopathological Images