Fast Convolutional Nets With fbfft: A GPU Performance Evaluation

Abstract

We examine the performance profile of Convolutional Neural Network training on the current generation of NVIDIA Graphics Processing Units. We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA's cuFFT library, and another based on a Facebook authored FFT implementation, fbfft, that provides significant speedups over cuFFT (over 1.5x) for whole CNNs. Both of these convolution implementations are available in open source, and are faster than NVIDIA's cuDNN implementation for many common convolutional layers (up to 23.5x for some synthetic kernel configurations). We discuss different performance regimes of convolutions, comparing areas where straightforward time domain convolutions outperform Fourier frequency domain convolutions. Details on algorithmic applications of NVIDIA GPU hardware specifics in the implementation of fbfft are also provided.

Camera ready for ICLR2015

References in corpus (3)

• Natural Language Processing (almost) from Scratch
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Fast Training of Convolutional Networks through FFTs

Cited by in corpus (48)

• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Object Detection in 20 Years: A Survey
• Channel Pruning for Accelerating Very Deep Neural Networks
• Recent Advances in Convolutional Neural Networks
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Deep Learning in the Automotive Industry: Applications and Tools
• Comparative Study of Deep Learning Software Frameworks
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• MEC: Memory-efficient Convolution for Deep Neural Network
• ZNN - A Fast and Scalable Algorithm for Training 3D Convolutional Networks on Multi-Core and Many-Core Shared Memory Machines
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• High Performance Zero-Memory Overhead Direct Convolutions
• Compressing Convolutional Neural Networks
• The Indirect Convolution Algorithm
• Compact Convolutional Neural Network Cascade for Face Detection
• TTC: A Tensor Transposition Compiler for Multiple Architectures
• FFT Convolutions are Faster than Winograd on Modern CPUs, Here is Why
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
• cltorch: a Hardware-Agnostic Backend for the Torch Deep Neural Network Library, Based on OpenCL
• A detailed comparative study of open source deep learning frameworks
• Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
• Distributed Training Large-Scale Deep Architectures
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
• Sequence embeddings help to identify fraudulent cases in healthcare insurance
• Deep Tensor Convolution on Multicores
• TTC: A high-performance Compiler for Tensor Transpositions
• Reducing the Training Time of Neural Networks by Partitioning
• Convolutional Neural Network Simplification with Progressive Retraining
• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
• GPU Fast Convolution via the Overlap-and-Save Method in Shared Memory
• L3 Fusion: Fast Transformed Convolutions on CPUs
• Going Deeper in Frequency Convolutional Neural Network: A Theoretical Perspective
• Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains
• ZNNi - Maximizing the Inference Throughput of 3D Convolutional Networks on Multi-Core CPUs and GPUs
• DWM: A Decomposable Winograd Method for Convolution Acceleration
• Design Light-weight 3D Convolutional Networks for Video Recognition Temporal Residual, Fully Separable Block, and Fast Algorithm
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• Exploiting Non-Local Priors via Self-Convolution For Highly-Efficient Image Restoration
• Faster Convergence & Generalization in DNNs
• See far with TPNET: a Tile Processor and a CNN Symbiosis
• Face Recognition with Hybrid Efficient Convolution Algorithms on FPGAs
• Prune the Convolutional Neural Networks with Sparse Shrink
• clcNet: Improving the Efficiency of Convolutional Neural Network using Channel Local Convolutions
• Machine Learning enabled Spectrum Sharing in Dense LTE-U/Wi-Fi Coexistence Scenarios
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• OpEvo: An Evolutionary Method for Tensor Operator Optimization