Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks

Abstract

Traditional image-based survival prediction models rely on discriminative patch labeling which make those methods not scalable to extend to large datasets. Recent studies have shown Multiple Instance Learning (MIL) framework is useful for histopathological images when no annotations are available in classification task. Different to the current image-based survival models that limit to key patches or clusters derived from Whole Slide Images (WSIs), we propose Deep Attention Multiple Instance Survival Learning (DeepAttnMISL) by introducing both siamese MI-FCN and attention-based MIL pooling to efficiently learn imaging features from the WSI and then aggregate WSI-level information to patient-level. Attention-based aggregation is more flexible and adaptive than aggregation techniques in recent survival models. We evaluated our methods on two large cancer whole slide images datasets and our results suggest that the proposed approach is more effective and suitable for large datasets and has better interpretability in locating important patterns and features that contribute to accurate cancer survival predictions. The proposed framework can also be used to assess individual patient's risk and thus assisting in delivering personalized medicine. Codes are available at https://github.com/uta-smile/DeepAttnMISL_MEDIA.

arXiv comment22 pages, 13 figures, published in Medical Image Analysis 65, 101789

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Structured Self-attentive Sentence Embedding
• Language Modeling with Gated Convolutional Networks
• Deep Learning for Identifying Metastatic Breast Cancer
• Attention-based Deep Multiple Instance Learning
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Deep learning-based survival prediction for multiple cancer types using histopathology images
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification
• Boosting the concordance index for survival data - a unified framework to derive and evaluate biomarker combinations
• Weakly Supervised Deep Learning for Thoracic Disease Classification and Localization on Chest X-rays
• Cancer Metastasis Detection With Neural Conditional Random Field

Cited by in corpus (15)

• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification
• AdvMIL: Adversarial Multiple Instance Learning for the Survival Analysis on Whole-Slide Images
• Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology
• DSCA: A Dual-Stream Network with Cross-Attention on Whole-Slide Image Pyramids for Cancer Prognosis
• PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning
• Colorectal cancer survival prediction using deep distribution based multiple-instance learning
• Self-Contrastive Weakly Supervised Learning Framework for Prognostic Prediction Using Whole Slide Images
• SCMIL: Sparse Context-aware Multiple Instance Learning for Predicting Cancer Survival Probability Distribution in Whole Slide Images
• Non-I.I.D. Multi-Instance Learning for Predicting Instance and Bag Labels using Variational Auto-Encoder
• Spectral Survival Analysis
• AMMASurv: Asymmetrical Multi-Modal Attention for Accurate Survival Analysis with Whole Slide Images and Gene Expression Data
• Finding Regions of Interest in Whole Slide Images Using Multiple Instance Learning
• MMAP: A Multi-Magnification and Prototype-Aware Architecture for Predicting Spatial Gene Expression
• DeepPrognosis: Preoperative Prediction of Pancreatic Cancer Survival and Surgical Margin via Contrast-Enhanced CT Imaging