Convolutional Neural Networks: Ensemble Modeling, Fine-Tuning and Unsupervised Semantic Localization for Intraoperative CLE Images

Abstract

Confocal laser endomicroscopy (CLE) is an advanced optical fluorescence technology undergoing assessment for applications in brain tumor surgery. Despite its promising potential, interpreting the unfamiliar gray tone images of fluorescent stains can be difficult. Many of the CLE images can be distorted by motion, extremely low or high fluorescence signal, or obscured by red blood cell accumulation, and these can be interpreted as nondiagnostic. However, just one neat CLE image might suffice for intraoperative diagnosis of the tumor. While manual examination of thousands of nondiagnostic images during surgery would be impractical, this creates an opportunity for a model to select diagnostic images for the pathologists or surgeon's review. In this study, we sought to develop a deep learning model to automatically detect the diagnostic images using a manually annotated dataset, and we employed a patient-based nested cross-validation approach to explore generalizability of the model. We explored various training regimes: deep training, shallow fine-tuning, and deep fine-tuning. Further, we investigated the effect of ensemble modeling by combining the top-5 single models crafted in the development phase. We localized histological features from diagnostic CLE images by visualization of shallow and deep neural activations. Our inter-rater experiment results confirmed that our ensemble of deeply fine-tuned models achieved higher agreement with the ground truth than the other observers. With the speed and precision of the proposed method (110 images/second; 85% on the gold standard test subset), it has potential to be integrated into the operative workflow in the brain tumor surgery.

The related work was updated

References in corpus (8)

• A Survey on Deep Learning in Medical Image Analysis
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis
• Deep Multi-scale Location-aware 3D Convolutional Neural Networks for Automated Detection of Lacunes of Presumed Vascular Origin
• Improving utility of brain tumor confocal laser endomicroscopy: objective value assessment and diagnostic frame detection with convolutional neural networks
• Auto-context Convolutional Neural Network (Auto-Net) for Brain Extraction in Magnetic Resonance Imaging

Cited by in corpus (7)

• Image computing for fibre-bundle endomicroscopy: A review
• Deep Transfer Learning Methods for Colon Cancer Classification in Confocal Laser Microscopy Images
• Prospects for Theranostics in Neurosurgical Imaging: Empowering Confocal Laser Endomicroscopy Diagnostics via Deep Learning
• TKD: Temporal Knowledge Distillation for Active Perception
• Evaluating Convolutional Neural Networks for COVID-19 classification in chest X-ray images
• Weakly-Supervised Learning-Based Feature Localization in Confocal Laser Endomicroscopy Glioma Images
• Fluorescence Image Histology Pattern Transformation using Image Style Transfer