MedViT: A Robust Vision Transformer for Generalized Medical Image Classification

Abstract

Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism while jointly attending to information in various representation subspaces, we construct our attention mechanism by means of an efficient convolution operation. Moreover, to alleviate the fragility of our Transformer model against adversarial attacks, we attempt to learn smoother decision boundaries. To this end, we augment the shape information of an image in the high-level feature space by permuting the feature mean and variance within mini-batches. With less computational complexity, our proposed hybrid model demonstrates its high robustness and generalization ability compared to the state-of-the-art studies on a large-scale collection of standardized MedMNIST-2D datasets.

References in corpus (19)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• Explaining and Harnessing Adversarial Examples
• The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions
• ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data
• How Do Vision Transformers Work?
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Smooth Adversarial Training
• Towards Evaluating the Robustness of Deep Diagnostic Models by Adversarial Attack
• On the Adversarial Robustness of Vision Transformers
• Adversarially Robust Neural Architectures
• Do Wider Neural Networks Really Help Adversarial Robustness?
• An Effective Anti-Aliasing Approach for Residual Networks
• A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability

Cited by in corpus (20)

• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Enhancing Skin Disease Classification Leveraging Transformer-based Deep Learning Architectures and Explainable AI
• Artificial General Intelligence for Medical Imaging Analysis
• Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention
• CoTCoNet: An Optimized Coupled Transformer-Convolutional Network with an Adaptive Graph Reconstruction for Leukemia Detection
• Brain Stroke Detection and Classification Using CT Imaging with Transformer Models and Explainable AI
• MIAFEx: An Attention-based Feature Extraction Method for Medical Image Classification
• Recent Advances in Medical Image Classification
• Improving Robustness and Reliability in Medical Image Classification with Latent-Guided Diffusion and Nested-Ensembles
• RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training
• Self Pre-training with Adaptive Mask Autoencoders for Variable-Contrast 3D Medical Imaging
• H-SemiS: Hierarchical Fusion of Semi and Self-Supervised Learning for Knee Osteoarthritis Severity Grading
• Efficient Lung Ultrasound Severity Scoring Using Dedicated Feature Extractor
• The Application of Deep Learning for Lymph Node Segmentation: A Systematic Review
• MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-Modal Masked Autoencoder
• Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data
• MedSymmFlow: Bridging Generative Modeling and Classification in Medical Imaging through Symmetrical Flow Matching
• Masked strategies for images with small objects
• Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis