Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates

Abstract

Large, fine-grained image segmentation datasets, annotated at pixel-level, are difficult to obtain, particularly in medical imaging, where annotations also require expert knowledge. Weakly-supervised learning can train models by relying on weaker forms of annotation, such as scribbles. Here, we learn to segment using scribble annotations in an adversarial game. With unpaired segmentation masks, we train a multi-scale GAN to generate realistic segmentation masks at multiple resolutions, while we use scribbles to learn their correct position in the image. Central to the model's success is a novel attention gating mechanism, which we condition with adversarial signals to act as a shape prior, resulting in better object localization at multiple scales. Subject to adversarial conditioning, the segmentor learns attention maps that are semantic, suppress the noisy activations outside the objects, and reduce the vanishing gradient problem in the deeper layers of the segmentor. We evaluated our model on several medical (ACDC, LVSC, CHAOS) and non-medical (PPSS) datasets, and we report performance levels matching those achieved by models trained with fully annotated segmentation masks. We also demonstrate extensions in a variety of settings: semi-supervised learning; combining multiple scribble sources (a crowdsourcing scenario) and multi-task learning (combining scribble and mask supervision). We release expert-made scribble annotations for the ACDC dataset, and the code used for the experiments, at https://vios-s.github.io/multiscale-adversarial-attention-gates

arXiv commentPaper accepted for publication at: IEEE Transaction on Medical Imaging - Project page: https://vios-s.github.io/multiscale-adversarial-attention-gates

References in corpus (11)

• Conditional Random Fields as Recurrent Neural Networks
• Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation
• SegAN: Adversarial Network with Multi-scale $L_1$ Loss for Medical Image Segmentation
• Constrained-CNN losses for weakly supervised segmentation
• Learning a Probabilistic Model for Diffeomorphic Registration
• Disentangled Representation Learning in Cardiac Image Analysis
• Amortised MAP Inference for Image Super-resolution
• Cardiac Segmentation with Strong Anatomical Guarantees
• Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation
• Semi-supervised few-shot learning for medical image segmentation
• Temporal Consistency Objectives Regularize the Learning of Disentangled Representations

Cited by in corpus (6)

• WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• PyMIC: A deep learning toolkit for annotation-efficient medical image segmentation
• QMaxViT-Unet+: A Query-Based MaxViT-Unet with Edge Enhancement for Scribble-Supervised Segmentation of Medical Images
• OCTAve: 2D en face Optical Coherence Tomography Angiography Vessel Segmentation in Weakly-Supervised Learning with Locality Augmentation
• How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model