Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture

Abstract

Histopathologic diagnosis relies on simultaneous integration of information from a broad range of scales, ranging from nuclear aberrations ($\approx \mathcal{O}(0.1\mu m)$) through cellular structures ($\approx \mathcal{O}(10\mu m)$) to the global tissue architecture ($⪆\mathcal{O}(1mm)$). To explicitly mimic how human pathologists combine multi-scale information, we introduce a family of multi-encoder FCNs with deep fusion. We present a simple block for merging model paths with differing spatial scales in a spatial relationship-preserving fashion, which can readily be included in standard encoder-decoder networks. Additionally, a context classification gate block is proposed as an alternative for the incorporation of global context. Our experiments were performed on three publicly available whole-slide images of recent challenges (PAIP 2019, BACH 2020, CAMELYON 2016). The multi-scale architectures consistently outperformed the baseline single-scale U-Nets by a large margin. They benefit from local as well as global context and particularly a combination of both. If feature maps from different scales are fused, doing so in a manner preserving spatial relationships was found to be beneficial. Deep guidance by a context classification loss appeared to improve model training at low computational costs. All multi-scale models had a reduced GPU memory footprint compared to ensembles of individual U-Nets trained on different image scales. Additional path fusions were shown to be possible at low computational cost, opening up possibilities for further, systematic and task-specific architecture optimization. The findings demonstrate the potential of the presented family of human-inspired, end-to-end trainable, multi-scale multi-encoder FCNs to improve deep histopathologic diagnosis by extensive integration of largely different spatial scales.

arXiv commentAccepted for Medical Image Analysis

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• A Survey on Deep Learning in Medical Image Analysis
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Deep Residual Learning for Image Recognition
• Attention U-Net: Learning Where to Look for the Pancreas
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• ParseNet: Looking Wider to See Better
• BACH: Grand Challenge on Breast Cancer Histology Images
• Detecting Cancer Metastases on Gigapixel Pathology Images
• nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
• RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification
• Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard
• The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
• Bottleneck Supervised U-Net for Pixel-wise Liver and Tumor Segmentation
• Multiscale Deep Neural Networks for Multiclass Tissue Classification of Histological Whole-Slide Images

Cited by in corpus (6)

• Nuclei Segmentation with Point Annotations from Pathology Images via Self-Supervised Learning and Co-Training
• A Dual-branch Self-supervised Representation Learning Framework for Tumour Segmentation in Whole Slide Images
• Cross-attention Spatio-temporal Context Transformer for Semantic Segmentation of Historical Maps
• A State-of-the-art Survey of Artificial Neural Networks for Whole-slide Image Analysis:from Popular Convolutional Neural Networks to Potential Visual Transformers
• Structure-aware scale-adaptive networks for cancer segmentation in whole-slide images
• Efficient Pyramidal Analysis of Gigapixel Images on a Decentralized Modest Computer Cluster