ZNN - A Fast and Scalable Algorithm for Training 3D Convolutional Networks on Multi-Core and Many-Core Shared Memory Machines

Abstract

Convolutional networks (ConvNets) have become a popular approach to computer vision. It is important to accelerate ConvNet training, which is computationally costly. We propose a novel parallel algorithm based on decomposition into a set of tasks, most of which are convolutions or FFTs. Applying Brent's theorem to the task dependency graph implies that linear speedup with the number of processors is attainable within the PRAM model of parallel computation, for wide network architectures. To attain such performance on real shared-memory machines, our algorithm computes convolutions converging on the same node of the network with temporal locality to reduce cache misses, and sums the convergent convolution outputs via an almost wait-free concurrent method to reduce time spent in critical sections. We implement the algorithm with a publicly available software package called ZNN. Benchmarking with multi-core CPUs shows that ZNN can attain speedup roughly equal to the number of physical cores. We also show that ZNN can attain over 90x speedup on a many-core CPU (Xeon Phi Knights Corner). These speedups are achieved for network architectures with widths that are in common use. The task parallelism of the ZNN algorithm is suited to CPUs, while the SIMD parallelism of previous algorithms is compatible with GPUs. Through examples, we show that ZNN can be either faster or slower than certain GPU implementations depending on specifics of the network architecture, kernel sizes, and density and size of the output patch. ZNN may be less costly to develop and maintain, due to the relative ease of general-purpose CPU programming.

References in corpus (7)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Fully Convolutional Networks for Semantic Segmentation
• cuDNN: Efficient Primitives for Deep Learning
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Locally Scale-Invariant Convolutional Neural Networks
• Recursive Training of 2D-3D Convolutional Networks for Neuronal Boundary Detection
• The Potential of the Intel Xeon Phi for Supervised Deep Learning

Cited by in corpus (10)

• High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors
• Optimizing CNN Model Inference on CPUs
• Automatic Neuron Detection in Calcium Imaging Data Using Convolutional Networks
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• A Multi-Pass Approach to Large-Scale Connectomics
• Recursive Training of 2D-3D Convolutional Networks for Neuronal Boundary Detection
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• ZNNi - Maximizing the Inference Throughput of 3D Convolutional Networks on Multi-Core CPUs and GPUs
• NUMA-aware FFT-based Convolution on ARMv8 Many-core CPUs