Recent Progress in Transformer-based Medical Image Analysis

Abstract

The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.

arXiv commentComputers in Biology and Medicine Accepted

References in corpus (48)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• mixup: Beyond Empirical Risk Minimization
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions
• ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
• Bootstrap your own latent: A new approach to self-supervised Learning
• Can AI help in screening Viral and COVID-19 pneumonia?
• Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)
• Low Dose CT Image Denoising Using a Generative Adversarial Network with Wasserstein Distance and Perceptual Loss
• A Structured Self-attentive Sentence Embedding
• The Medical Segmentation Decathlon
• A Deep Reinforced Model for Abstractive Summarization
• MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification
• Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
• Training Generative Adversarial Networks with Limited Data
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• Linformer: Self-Attention with Linear Complexity
• REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs
• CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation
• COVID-19 Image Data Collection
• TransMorph: Transformer for unsupervised medical image registration
• COVID-CT-Dataset: A CT Scan Dataset about COVID-19
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Reliable Tuberculosis Detection using Chest X-ray with Deep Learning, Segmentation and Visualization
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• Can Deep Learning Outperform Modern Commercial CT Image Reconstruction Methods?
• Multivariate mixture model for myocardium segmentation combining multi-source images
• VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images
• Med3D: Transfer Learning for 3D Medical Image Analysis
• The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes
• Escaping the Big Data Paradigm with Compact Transformers
• A Recurrent CNN for Automatic Detection and Classification of Coronary Artery Plaque and Stenosis in Coronary CT Angiography
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• Constrained Deep Weak Supervision for Histopathology Image Segmentation
• Attention Enriched Deep Learning Model for Breast Tumor Segmentation in Ultrasound Images
• PanNuke Dataset Extension, Insights and Baselines
• BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset
• 2018 Robotic Scene Segmentation Challenge
• BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients
• VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Deep Retinal Image Segmentation with Regularization Under Geometric Priors
• Transformers in Medical Image Analysis: A Review
• A Transformer-based Generative Adversarial Network for Brain Tumor Segmentation
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• Data-Efficient Vision Transformers for Multi-Label Disease Classification on Chest Radiographs

Cited by in corpus (3)

• Enhancing Skin Disease Classification Leveraging Transformer-based Deep Learning Architectures and Explainable AI
• Segmenting Medical Images with Limited Data
• UNGT: Ultrasound Nasogastric Tube Dataset for Medical Image Analysis