Fast Training of Convolutional Networks through FFTs

Abstract

Convolutional networks are one of the most widely employed architectures in computer vision and machine learning. In order to leverage their ability to learn complex functions, large amounts of data are required for training. Training a large convolutional network to produce state-of-the-art results can take weeks, even when using modern GPUs. Producing labels using a trained network can also be costly when dealing with web-scale datasets. In this work, we present a simple algorithm which accelerates training and inference by a significant factor, and can yield improvements of over an order of magnitude compared to existing state-of-the-art implementations. This is done by computing convolutions as pointwise products in the Fourier domain while reusing the same transformed feature map many times. The algorithm is implemented on a GPU architecture and addresses a number of related challenges.

Cited by in corpus (89)

• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Deep Convolutional Networks on Graph-Structured Data
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Fourier Neural Operator for Parametric Partial Differential Equations
• cuDNN: Efficient Primitives for Deep Learning
• Compressing Deep Convolutional Networks using Vector Quantization
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Pruning Filters for Efficient ConvNets
• Object Detection in 20 Years: A Survey
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Channel Pruning for Accelerating Very Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Recent Advances in Convolutional Neural Networks
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Pooling Methods in Deep Neural Networks, a Review
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• Origami: A 803 GOp/s/W Convolutional Network Accelerator
• Dynamic Network Surgery for Efficient DNNs
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Comparative Study of Deep Learning Software Frameworks
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Fast Algorithms for Convolutional Neural Networks
• Highly Efficient Forward and Backward Propagation of Convolutional Neural Networks for Pixelwise Classification
• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
• BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• A Survey on Deep Learning Methods for Robot Vision
• MEC: Memory-efficient Convolution for Deep Neural Network
• Building Efficient ConvNets using Redundant Feature Pruning
• An exploration of parameter redundancy in deep networks with circulant projections
• Enabling Sparse Winograd Convolution by Native Pruning
• High Performance Zero-Memory Overhead Direct Convolutions
• Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• Compressing Convolutional Neural Networks
• On Vectorization of Deep Convolutional Neural Networks for Vision Tasks
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Fast ConvNets Using Group-wise Brain Damage
• FFT Convolutions are Faster than Winograd on Modern CPUs, Here is Why
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Searching for Winograd-aware Quantized Networks
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
• A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-Time
• Cuttlefish: A Lightweight Primitive for Adaptive Query Processing
• Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
• Spatial-Winograd Pruning Enabling Sparse Winograd Convolution
• Can we integrate spatial verification methods into neural-network loss functions for atmospheric science?
• Deep Networks from the Principle of Rate Reduction
• Distributed Training Large-Scale Deep Architectures
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
• SpecNet: Spectral Domain Convolutional Neural Network
• Deep Tensor Convolution on Multicores
• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
• FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation
• Frequency Domain Convolutional Neural Network: Accelerated CNN for Large Diabetic Retinopathy Image Classification
• FFT-Based Deep Learning Deployment in Embedded Systems
• Going Deeper in Frequency Convolutional Neural Network: A Theoretical Perspective
• Fast Convolution based on Winograd Minimum Filtering: Introduction and Development
• An Efficient Neighborhood-based Interaction Model for Recommendation on Heterogeneous Graph
• WSNet: Compact and Efficient Networks Through Weight Sampling
• Compression of Deep Neural Networks on the Fly
• Communication Lower Bound in Convolution Accelerators
• Recent Advances in Convolutional Neural Network Acceleration
• Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains
• ThumbNet: One Thumbnail Image Contains All You Need for Recognition
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• Faster Convergence & Generalization in DNNs
• LANCE: Efficient Low-Precision Quantized Winograd Convolution for Neural Networks Based on Graphics Processing Units
• See far with TPNET: a Tile Processor and a CNN Symbiosis
• CircConv: A Structured Convolution with Low Complexity
• Building Fast and Compact Convolutional Neural Networks for Offline Handwritten Chinese Character Recognition
• Exploiting Non-Local Priors via Self-Convolution For Highly-Efficient Image Restoration
• Radius Adaptive Convolutional Neural Network
• clcNet: Improving the Efficiency of Convolutional Neural Network using Channel Local Convolutions
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• Fast Intent Classification for Spoken Language Understanding
• OpEvo: An Evolutionary Method for Tensor Operator Optimization
• Fixed-point Factorized Networks
• Sequence Prediction using Spectral RNNs
• Efficient Residue Number System Based Winograd Convolution
• Eigen component analysis: A quantum theory incorporated machine learning technique to find linearly maximum separable components
• Accelerating convolutional neural network by exploiting sparsity on GPUs