Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation

Abstract

Convolutional Neural Networks (CNNs) can be shifted across 2D images or 3D videos to segment them. They have a fixed input size and typically perceive only small local contexts of the pixels to be classified as foreground or background. In contrast, Multi-Dimensional Recurrent NNs (MD-RNNs) can perceive the entire spatio-temporal context of each pixel in a few sweeps through all pixels, especially when the RNN is a Long Short-Term Memory (LSTM). Despite these theoretical advantages, however, unlike CNNs, previous MD-LSTM variants were hard to parallelize on GPUs. Here we re-arrange the traditional cuboid order of computations in MD-LSTM in pyramidal fashion. The resulting PyraMiD-LSTM is easy to parallelize, especially for 3D data such as stacks of brain slice images. PyraMiD-LSTM achieved best known pixel-wise brain image segmentation results on MRBrainS13 (and competitive results on EM-ISBI12).

Marijn F. Stollenga and Wonmin Byeon are the shared first authors, both authors contributed equally to this work

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Natural Scales in Geographical Patterns
• Recurrent Neural Network Regularization
• cuDNN: Efficient Primitives for Deep Learning

Cited by in corpus (20)

• A Survey on Deep Learning in Medical Image Analysis
• Pixel Recurrent Neural Networks
• Applications of Deep Learning and Reinforcement Learning to Biological Data
• A Review on Deep Learning Techniques for Video Prediction
• FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics
• Speech Emotion Recognition with Dual-Sequence LSTM Architecture
• A Review of Hidden Markov Models and Recurrent Neural Networks for Event Detection and Localization in Biomedical Signals
• Upper Esophageal Sphincter Opening Segmentation with Convolutional Recurrent Neural Networks in High Resolution Cervical Auscultation
• Quantitative analysis of patch-based fully convolutional neural networks for tissue segmentation on brain magnetic resonance imaging
• DeepHealth: Review and challenges of artificial intelligence in health informatics
• Neural Attribute Machines for Program Generation
• Deep Learning in Mining Biological Data
• Fast Fourier Transformation for Optimizing Convolutional Neural Networks in Object Recognition
• DartsReNet: Exploring new RNN cells in ReNet architectures
• Deep Learning: Our Miraculous Year 1990-1991
• Recurrent neural networks for aortic image sequence segmentation with sparse annotations
• Cubic LSTMs for Video Prediction
• Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving
• ARMA Nets: Expanding Receptive Field for Dense Prediction
• Veritatem Dies Aperit- Temporally Consistent Depth Prediction Enabled by a Multi-Task Geometric and Semantic Scene Understanding Approach