TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

Abstract

Medical image segmentation is an essential prerequisite for developing healthcare systems, especially for disease diagnosis and treatment planning. On various medical image segmentation tasks, the u-shaped architecture, also known as U-Net, has become the de-facto standard and achieved tremendous success. However, due to the intrinsic locality of convolution operations, U-Net generally demonstrates limitations in explicitly modeling long-range dependency. Transformers, designed for sequence-to-sequence prediction, have emerged as alternative architectures with innate global self-attention mechanisms, but can result in limited localization abilities due to insufficient low-level details. In this paper, we propose TransUNet, which merits both Transformers and U-Net, as a strong alternative for medical image segmentation. On one hand, the Transformer encodes tokenized image patches from a convolution neural network (CNN) feature map as the input sequence for extracting global contexts. On the other hand, the decoder upsamples the encoded features which are then combined with the high-resolution CNN feature maps to enable precise localization. We argue that Transformers can serve as strong encoders for medical image segmentation tasks, with the combination of U-Net to enhance finer details by recovering localized spatial information. TransUNet achieves superior performances to various competing methods on different medical applications including multi-organ segmentation and cardiac segmentation. Code and models are available at https://github.com/Beckschen/TransUNet.

arXiv comment13 pages, 3 figures

References in corpus (4)

• Attention U-Net: Learning Where to Look for the Pancreas
• Generating Long Sequences with Sparse Transformers
• Automatic 3D Cardiovascular MR Segmentation with Densely-Connected Volumetric ConvNets
• Domain Adaptive Relational Reasoning for 3D Multi-Organ Segmentation

Cited by in corpus (144)

• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• Medical Image Segmentation Using Deep Learning: A Survey
• TransMorph: Transformer for unsupervised medical image registration
• A General Survey on Attention Mechanisms in Deep Learning
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection
• Video-SwinUNet: Spatio-temporal Deep Learning Framework for VFSS Instance Segmentation
• DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• nnFormer: Interleaved Transformer for Volumetric Segmentation
• MTU-Net: Multi-level TransUNet for Space-based Infrared Tiny Ship Detection
• UNETR: Transformers for 3D Medical Image Segmentation
• Dual Cross-Attention for Medical Image Segmentation
• Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
• SCTransNet: Spatial-channel Cross Transformer Network for Infrared Small Target Detection
• A Survey on Deep Learning for Skin Lesion Segmentation
• CSWin-UNet: Transformer UNet with Cross-Shaped Windows for Medical Image Segmentation
• FCN-Transformer Feature Fusion for Polyp Segmentation
• Transformers in Healthcare: A Survey
• WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image
• MISSFormer: An Effective Medical Image Segmentation Transformer
• TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
• Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images
• ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration
• CaraNet: Context Axial Reverse Attention Network for Segmentation of Small Medical Objects
• CMU-Net: A Strong ConvMixer-based Medical Ultrasound Image Segmentation Network
• Boundary-aware Transformers for Skin Lesion Segmentation
• Recent Progress in Transformer-based Medical Image Analysis
• TransBTS: Multimodal Brain Tumor Segmentation Using Transformer
• Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection
• TranSMS: Transformers for Super-Resolution Calibration in Magnetic Particle Imaging
• RCPS: Rectified Contrastive Pseudo Supervision for Semi-Supervised Medical Image Segmentation
• BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once
• Is it Time to Replace CNNs with Transformers for Medical Images?
• TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation
• Vision Transformers, a new approach for high-resolution and large-scale mapping of canopy heights
• TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation
• A Convolutional-Transformer Network for Crack Segmentation with Boundary Awareness
• MicroSegNet: A Deep Learning Approach for Prostate Segmentation on Micro-Ultrasound Images
• A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation
• Virtual histological staining of unlabeled autopsy tissue
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis
• HiDiff: Hybrid Diffusion Framework for Medical Image Segmentation
• Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• Multiscale Vision Transformers
• LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• FDiff-Fusion:Denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation
• A Transformer-based Generative Adversarial Network for Brain Tumor Segmentation
• HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images
• A Survey of Visual Transformers
• Pyramid Medical Transformer for Medical Image Segmentation
• Optimizing Vision Transformers for Medical Image Segmentation
• Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples
• SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation
• Glance-and-Gaze Vision Transformer
• CiT-Net: Convolutional Neural Networks Hand in Hand with Vision Transformers for Medical Image Segmentation
• Prototype Learning Guided Hybrid Network for Breast Tumor Segmentation in DCE-MRI
• Concurrent ischemic lesion age estimation and segmentation of CT brain using a Transformer-based network
• CAVE: Cerebral Artery-Vein Segmentation in Digital Subtraction Angiography
• QMaxViT-Unet+: A Query-Based MaxViT-Unet with Edge Enhancement for Scribble-Supervised Segmentation of Medical Images
• Infrared Small-Dim Target Detection with Transformer under Complex Backgrounds
• Semantic Labeling of High Resolution Images Using EfficientUNets and Transformers
• Transformer-Unet: Raw Image Processing with Unet
• Complex Network for Complex Problems: A comparative study of CNN and Complex-valued CNN
• Deep Learning-based Bathymetry Retrieval without In-situ Depths using Remote Sensing Imagery and SfM-MVS DSMs with Data Gaps
• Medical Image Segmentation Using Squeeze-and-Expansion Transformers
• Salient Mask-Guided Vision Transformer for Fine-Grained Classification
• SKA Science Data Challenge 2: analysis and results
• DnSwin: Toward Real-World Denoising via Continuous Wavelet Sliding-Transformer
• Exploring Multilingual Large Language Models for Enhanced TNM classification of Radiology Report in lung cancer staging
• Transformer-Based Source-Free Domain Adaptation
• Improving Across-Dataset Brain Tissue Segmentation Using Transformer
• MLN-net: A multi-source medical image segmentation method for clustered microcalcifications using multiple layer normalization
• EDMAE: An Efficient Decoupled Masked Autoencoder for Standard View Identification in Pediatric Echocardiography
• SF2Former: Amyotrophic Lateral Sclerosis Identification From Multi-center MRI Data Using Spatial and Frequency Fusion Transformer
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation
• HSMix: Hard and Soft Mixing Data Augmentation for Medical Image Segmentation
• DSCA: A Digital Subtraction Angiography Sequence Dataset and Spatio-Temporal Model for Cerebral Artery Segmentation
• MUCM-Net: A Mamba Powered UCM-Net for Skin Lesion Segmentation
• Towards Safer Robot-Assisted Surgery: A Markerless Augmented Reality Framework
• HST-MRF: Heterogeneous Swin Transformer with Multi-Receptive Field for Medical Image Segmentation
• A Cascade Transformer-based Model for 3D Dose Distribution Prediction in Head and Neck Cancer Radiotherapy
• Automated Measurement of Vascular Calcification in Femoral Endarterectomy Patients Using Deep Learning
• Multi-Compound Transformer for Accurate Biomedical Image Segmentation
• Small but Mighty: Enhancing 3D Point Clouds Semantic Segmentation with U-Next Framework
• Neural Transformers for Intraductal Papillary Mucosal Neoplasms (IPMN) Classification in MRI images
• GiT: Graph Interactive Transformer for Vehicle Re-identification
• Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Axial multi-layer perceptron architecture for automatic segmentation of choroid plexus in multiple sclerosis
• Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation
• PECI-Net: Bolus segmentation from video fluoroscopic swallowing study images using preprocessing ensemble and cascaded inference
• TransCT: Dual-path Transformer for Low Dose Computed Tomography
• CB-HVTNet: A channel-boosted hybrid vision transformer network for lymphocyte assessment in histopathological images
• A Multi-Branch Hybrid Transformer Networkfor Corneal Endothelial Cell Segmentation
• SDNet: mutil-branch for single image deraining using swin
• Leveraging Labelled Data Knowledge: A Cooperative Rectification Learning Network for Semi-supervised 3D Medical Image Segmentation
• SeqSeg: Learning Local Segments for Automatic Vascular Model Construction
• Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation
• Advances in Medical Image Segmentation: A Comprehensive Survey with a Focus on Lumbar Spine Applications
• A Comprehensive Study on Medical Image Segmentation using Deep Neural Networks
• Recent Advances in Medical Image Classification
• Hepatic vessel segmentation based on 3D swin-transformer with inductive biased multi-head self-attention
• Pediatric Pancreas Segmentation from MRI Scans with Deep Learning
• More than Encoder: Introducing Transformer Decoder to Upsample
• Retinal OCT Synthesis with Denoising Diffusion Probabilistic Models for Layer Segmentation
• PlaNet-S: Automatic Semantic Segmentation of Placenta
• Are Vision xLSTM Embedded UNet More Reliable in Medical 3D Image Segmentation?
• Toward Accurate and Reliable Iris Segmentation Using Uncertainty Learning
• 3D Distance-color-coded Assessment of PCI Stent Apposition via Deep-learning-based Three-dimensional Multi-object Segmentation
• Advanced Deep Learning Architectures for Accurate Detection of Subsurface Tile Drainage Pipes from Remote Sensing Images
• A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers
• Synthesis in Style: Semantic Segmentation of Historical Documents using Synthetic Data
• Accurate Lung Nodules Segmentation with Detailed Representation Transfer and Soft Mask Supervision
• SA${}^2$Net: Scale-Adaptive Structure-Affinity Transformation for Spine Segmentation from Ultrasound Volume Projection Imaging
• BreakNet: Discontinuity-Resilient Multi-Scale Transformer Segmentation of Retinal Layers
• Weakly Supervised Lymph Nodes Segmentation Based on Partial Instance Annotations with Pre-trained Dual-branch Network and Pseudo Label Learning
• Cross-Domain Transfer Learning with CoRTe: Consistent and Reliable Transfer from Black-Box to Lightweight Segmentation Model
• Universal Design Methodology for Printable Microstructural Materials via a New Deep Generative Learning Model: Application to a Piezocomposite
• Transformation Invariant Cancerous Tissue Classification Using Spatially Transformed DenseNet
• RadioNet: Transformer based Radio Map Prediction Model For Dense Urban Environments
• RTA-Former: Reverse Transformer Attention for Polyp Segmentation
• Sequential Learning on Liver Tumor Boundary Semantics and Prognostic Biomarker Mining
• CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning
• TransMix: Attend to Mix for Vision Transformers
• Dual Interaction Network with Cross-Image Attention for Medical Image Segmentation
• Dispensed Transformer Network for Unsupervised Domain Adaptation
• LambdaUNet: 2.5D Stroke Lesion Segmentation of Diffusion-weighted MR Images
• Combining CNNs With Transformer for Multimodal 3D MRI Brain Tumor Segmentation With Self-Supervised Pretraining
• Investigating and Improving Latent Density Segmentation Models for Aleatoric Uncertainty Quantification in Medical Imaging
• VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classification
• Improving Lesion Segmentation in Medical Images by Global and Regional Feature Compensation
• UltraPose: Synthesizing Dense Pose with 1 Billion Points by Human-body Decoupling 3D Model
• SegAssess: Panoramic quality mapping for robust and transferable unsupervised segmentation assessment
• ST-DETR: Spatio-Temporal Object Traces Attention Detection Transformer
• DeepFDR: A Deep Learning-based False Discovery Rate Control Method for Neuroimaging Data
• SFB-net for cardiac segmentation: Bridging the semantic gap with attention
• Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• Improving FHB Screening in Wheat Breeding Using an Efficient Transformer Model
• Muscle volume quantification: guiding transformers with anatomical priors