NiftyNet: a deep-learning platform for medical imaging

Abstract

Medical image analysis and computer-assisted intervention problems are increasingly being addressed with deep-learning-based solutions. Established deep-learning platforms are flexible but do not provide specific functionality for medical image analysis and adapting them for this application requires substantial implementation effort. Thus, there has been substantial duplication of effort and incompatible infrastructure developed across many research groups. This work presents the open-source NiftyNet platform for deep learning in medical imaging. The ambition of NiftyNet is to accelerate and simplify the development of these solutions, and to provide a common mechanism for disseminating research outputs for the community to use, adapt and build upon. NiftyNet provides a modular deep-learning pipeline for a range of medical imaging applications including segmentation, regression, image generation and representation learning applications. Components of the NiftyNet pipeline including data loading, data augmentation, network architectures, loss functions and evaluation metrics are tailored to, and take advantage of, the idiosyncracies of medical image analysis and computer-assisted intervention. NiftyNet is built on TensorFlow and supports TensorBoard visualization of 2D and 3D images and computational graphs by default. We present 3 illustrative medical image analysis applications built using NiftyNet: (1) segmentation of multiple abdominal organs from computed tomography; (2) image regression to predict computed tomography attenuation maps from brain magnetic resonance images; and (3) generation of simulated ultrasound images for specified anatomical poses. NiftyNet enables researchers to rapidly develop and distribute deep learning solutions for segmentation, regression, image generation and representation learning applications, or extend the platform to new applications.

Wenqi Li and Eli Gibson contributed equally to this work. M. Jorge Cardoso and Tom Vercauteren contributed equally to this work. 26 pages, 6 figures; Update includes additional applications, updated author list and formatting for journal submission

References in corpus (9)

• Conditional Generative Adversarial Nets
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Theano: new features and speed improvements
• cuDNN: Efficient Primitives for Deep Learning
• Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks
• On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Freehand Ultrasound Image Simulation with Spatially-Conditioned Generative Adversarial Networks

Cited by in corpus (66)

• An overview of deep learning in medical imaging focusing on MRI
• Recent advances and clinical applications of deep learning in medical image analysis
• CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation
• Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
• TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning
• Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks
• Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration
• Models Genesis
• Differential Privacy-enabled Federated Learning for Sensitive Health Data
• Automatic Brain Tumor Segmentation using Convolutional Neural Networks with Test-Time Augmentation
• MIScnn: A Framework for Medical Image Segmentation with Convolutional Neural Networks and Deep Learning
• Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
• CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
• Reinventing 2D Convolutions for 3D Images
• Generalised Wasserstein Dice Score for Imbalanced Multi-class Segmentation using Holistic Convolutional Networks
• DLTK: State of the Art Reference Implementations for Deep Learning on Medical Images
• PyMIC: A deep learning toolkit for annotation-efficient medical image segmentation
• Generalized Wasserstein Dice Score, Distributionally Robust Deep Learning, and Ranger for brain tumor segmentation: BraTS 2020 challenge
• Adversarial Deformation Regularization for Training Image Registration Neural Networks
• Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss
• medigan: a Python library of pretrained generative models for medical image synthesis
• DeepReg: a deep learning toolkit for medical image registration
• pymia: A Python package for data handling and evaluation in deep learning-based medical image analysis
• Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
• GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging
• SenseCare: A Research Platform for Medical Image Informatics and Interactive 3D Visualization
• Medical Transformer: Universal Brain Encoder for 3D MRI Analysis
• Generation of microbial colonies dataset with deep learning style transfer
• Uncertainty in multitask learning: joint representations for probabilistic MR-only radiotherapy planning
• Curriculum learning for improved femur fracture classification: scheduling data with prior knowledge and uncertainty
• Deep Learning-Guided Surface Characterization for Autonomous Hydrogen Lithography
• Task Decomposition and Synchronization for Semantic Biomedical Image Segmentation
• Automatic segmentation method of pelvic floor levator hiatus in ultrasound using a self-normalising neural network
• Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
• Improved MR to CT synthesis for PET/MR attenuation correction using Imitation Learning
• Physics-informed brain MRI segmentation
• An Unsupervised Approach to Ultrasound Elastography with End-to-end Strain Regularisation
• Conditional Segmentation in Lieu of Image Registration
• Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE
• IrisNet: Deep Learning for Automatic and Real-time Tongue Contour Tracking in Ultrasound Video Data using Peripheral Vision
• Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions
• Integrative Imaging Informatics for Cancer Research: Workflow Automation for Neuro-oncology (I3CR-WANO)
• Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration
• Deep learning-based attenuation correction in the image domain for myocardial perfusion SPECT imaging
• Fast Infant MRI Skullstripping with Multiview 2D Convolutional Neural Networks
• Deep Learning in Multi-organ Segmentation
• Accurate Automatic Segmentation of Amygdala Subnuclei and Modeling of Uncertainty via Bayesian Fully Convolutional Neural Network
• 3D Segmentation Networks for Excessive Numbers of Classes: Distinct Bone Segmentation in Upper Bodies
• A Decoupled Uncertainty Model for MRI Segmentation Quality Estimation
• Interactive Segmentation via Deep Learning and B-Spline Explicit Active Surfaces
• Fully Automatic Segmentation of 3D Brain Ultrasound: Learning from Coarse Annotations
• Hepatic vessel segmentation using a reduced filter 3D U-Net in ultrasound imaging
• MeDaS: An open-source platform as service to help break the walls between medicine and informatics
• Eisen: a python package for solid deep learning
• Quantitative Parametric Mapping of Tissues Properties from Standard Magnetic Resonance Imaging Enabled by Deep Learning
• Brain MR Image Segmentation in Small Dataset with Adversarial Defense and Task Reorganization
• Estimating MRI Image Quality via Image Reconstruction Uncertainty
• Deep Learning -- A first Meta-Survey of selected Reviews across Scientific Disciplines, their Commonalities, Challenges and Research Impact
• Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning
• PIMMS: Permutation Invariant Multi-Modal Segmentation
• Organ At Risk Segmentation with Multiple Modality
• Learning joint lesion and tissue segmentation from task-specific hetero-modal datasets
• Regional Deep Atrophy: a Self-Supervised Learning Method to Automatically Identify Regions Associated With Alzheimer's Disease Progression From Longitudinal MRI
• Spectral Data Augmentation Techniques to quantify Lung Pathology from CT-images
• CompareNet: Anatomical Segmentation Network with Deep Non-local Label Fusion
• Deep Boosted Regression for MR to CT Synthesis