CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

Abstract

Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a {\bf Co}nvolutional neural network and a {\bf Tr}ansformer {\bf (CoTr)} for accurate 3D medical image segmentation. Under this framework, the CNN is constructed to extract feature representations and an efficient deformable Transformer (DeTrans) is built to model the long-range dependency on the extracted feature maps. Different from the vanilla Transformer which treats all image positions equally, our DeTrans pays attention only to a small set of key positions by introducing the deformable self-attention mechanism. Thus, the computational and spatial complexities of DeTrans have been greatly reduced, making it possible to process the multi-scale and high-resolution feature maps, which are usually of paramount importance for image segmentation. We conduct an extensive evaluation on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that covers 11 major human organs. The results indicate that our CoTr leads to a substantial performance improvement over other CNN-based, transformer-based, and hybrid methods on the 3D multi-organ segmentation task. Code is available at \def\UrlFont{\rm\small\ttfamily} \url{https://github.com/YtongXie/CoTr}

arXiv comment13 pages

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• Pre-Trained Image Processing Transformer
• End-to-End Video Instance Segmentation with Transformers
• Inductive Bias of Deep Convolutional Networks through Pooling Geometry

Cited by in corpus (10)

• Recent advances and clinical applications of deep learning in medical image analysis
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• nnFormer: Interleaved Transformer for Volumetric Segmentation
• UNETR: Transformers for 3D Medical Image Segmentation
• MISSFormer: An Effective Medical Image Segmentation Transformer
• BiTr-Unet: a CNN-Transformer Combined Network for MRI Brain Tumor Segmentation
• Neural Transformers for Intraductal Papillary Mucosal Neoplasms (IPMN) Classification in MRI images
• Axial multi-layer perceptron architecture for automatic segmentation of choroid plexus in multiple sclerosis