TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

Abstract

Medical image segmentation - the prerequisite of numerous clinical needs - has been significantly prospered by recent advances in convolutional neural networks (CNNs). However, it exhibits general limitations on modeling explicit long-range relation, and existing cures, resorting to building deep encoders along with aggressive downsampling operations, leads to redundant deepened networks and loss of localized details. Hence, the segmentation task awaits a better solution to improve the efficiency of modeling global contexts while maintaining a strong grasp of low-level details. In this paper, we propose a novel parallel-in-branch architecture, TransFuse, to address this challenge. TransFuse combines Transformers and CNNs in a parallel style, where both global dependency and low-level spatial details can be efficiently captured in a much shallower manner. Besides, a novel fusion technique - BiFusion module is created to efficiently fuse the multi-level features from both branches. Extensive experiments demonstrate that TransFuse achieves the newest state-of-the-art results on both 2D and 3D medical image sets including polyp, skin lesion, hip, and prostate segmentation, with significant parameter decrease and inference speed improvement.

arXiv commentAccepted by MICCAI 2021

References in corpus (9)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Attention U-Net: Learning Where to Look for the Pancreas
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Road Extraction by Deep Residual U-Net
• Wide Residual Networks
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS
• SLSDeep: Skin Lesion Segmentation Based on Dilated Residual and Pyramid Pooling Networks

Cited by in corpus (8)

• Recent advances and clinical applications of deep learning in medical image analysis
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• nnFormer: Interleaved Transformer for Volumetric Segmentation
• UNETR: Transformers for 3D Medical Image Segmentation
• TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation
• A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends
• Pyramid Medical Transformer for Medical Image Segmentation
• Hepatic vessel segmentation based on 3D swin-transformer with inductive biased multi-head self-attention