Vision Transformers for Small Histological Datasets Learned through Knowledge Distillation
arXiv:2305.17370 · doi:10.1007/978-3-031-33380-4_13
Abstract
Computational Pathology (CPATH) systems have the potential to automate diagnostic tasks. However, the artifacts on the digitized histological glass slides, known as Whole Slide Images (WSIs), may hamper the overall performance of CPATH systems. Deep Learning (DL) models such as Vision Transformers (ViTs) may detect and exclude artifacts before running the diagnostic algorithm. A simple way to develop robust and generalized ViTs is to train them on massive datasets. Unfortunately, acquiring large medical datasets is expensive and inconvenient, prompting the need for a generalized artifact detection method for WSIs. In this paper, we present a student-teacher recipe to improve the classification performance of ViT for the air bubbles detection task. ViT, trained under the student-teacher framework, boosts its performance by distilling existing knowledge from the high-capacity teacher model. Our best-performing ViT yields 0.961 and 0.911 F1-score and MCC, respectively, observing a 7% gain in MCC against stand-alone training. The proposed method presents a new perspective of leveraging knowledge distillation over transfer learning to encourage the use of customized transformers for efficient preprocessing pipelines in the CPATH systems.
arXiv commentAccepted at PAKDD 2023
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Intriguing Properties of Vision Transformers
- MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
- Attention-based Clinical Note Summarization
- Detection and Localization of Melanoma Skin Cancer in Histopathological Whole Slide Images
- Multi-input segmentation of damaged brain in acute ischemic stroke patients using slow fusion with skip connection
Cited by in corpus (4)
- A survey of the Vision Transformers and their CNN-Transformer based Variants
- Detection and Localization of Melanoma Skin Cancer in Histopathological Whole Slide Images
- CT Perfusion is All We Need: 4D CNN Segmentation of Penumbra and Core in Patients With Suspected Ischemic Stroke
- Towards a privacy-preserving distributed cloud service for preprocessing very large medical images