nnFormer: Interleaved Transformer for Volumetric Segmentation

Abstract

Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to overcome their inherent shortcomings of spatial inductive bias. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer, a 3D transformer for volumetric medical image segmentation. nnFormer not only exploits the combination of interleaved convolution and self-attention operations, but also introduces local and global volume-based self-attention mechanism to learn volume representations. Moreover, nnFormer proposes to use skip attention to replace the traditional concatenation/summation operations in skip connections in U-Net like architecture. Experiments show that nnFormer significantly outperforms previous transformer-based counterparts by large margins on three public datasets. Compared to nnUNet, nnFormer produces significantly lower HD95 and comparable DSC results. Furthermore, we show that nnFormer and nnUNet are highly complementary to each other in model ensembling.

Journal version: more experiments and thorough comparison with nnUNet. Codes and models of nnFormer are available at https://git.io/JSf3i

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• Fast-SCNN: Fast Semantic Segmentation Network
• TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation
• SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation
• Medical Image Segmentation Using Squeeze-and-Expansion Transformers
• Are Convolutional Neural Networks or Transformers more like human vision?
• More than Encoder: Introducing Transformer Decoder to Upsample

Cited by in corpus (14)

• TransMorph: Transformer for unsupervised medical image registration
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• Dual Cross-Attention for Medical Image Segmentation
• Transformers in Healthcare: A Survey
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
• Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation
• Optimizing Vision Transformers for Medical Image Segmentation
• QMaxViT-Unet+: A Query-Based MaxViT-Unet with Edge Enhancement for Scribble-Supervised Segmentation of Medical Images
• DuDoTrans: Dual-Domain Transformer Provides More Attention for Sinogram Restoration in Sparse-View CT Reconstruction
• GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation
• Automated Measurement of Vascular Calcification in Femoral Endarterectomy Patients Using Deep Learning
• Error correcting 2D-3D cascaded network for myocardial infarct scar segmentation on late gadolinium enhancement cardiac magnetic resonance images
• Dual Interaction Network with Cross-Image Attention for Medical Image Segmentation
• SFB-net for cardiac segmentation: Bridging the semantic gap with attention