An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR

Abstract

In this project, we have successfully designed, implemented, deployed and tested a novel FPGA accelerated algorithm for neural network training. The algorithm itself was developed in an independent study option. This training method is based on Alternating Direction Method of Multipliers algorithm, which has strong parallel characteristics and avoids procedures such as matrix inversion that are problematic in hardware designs by employing LSMR. As an intermediate stage, we fully implemented the ADMM-LSMR method in C language for feed-forward neural networks with a flexible number of layers and hidden size. We demonstrated that the method can operate with fixed-point arithmetic without compromising the accuracy. Next, we devised an FPGA accelerated version of the algorithm using Intel FPGA SDK for OpenCL and performed extensive optimisation stages followed by successful deployment of the program on an Intel Arria 10 GX FPGA. The FPGA accelerated program showed up to 6 times speed up comparing to equivalent CPU implementation while achieving promising accuracy.

References in corpus (17)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Mixed Precision Training
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Ternary Weight Networks
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Training deep neural networks with low precision multiplications
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• High-Accuracy Low-Precision Training
• Training DNNs with Hybrid Block Floating Point
• SWALP : Stochastic Weight Averaging in Low-Precision Training
• A General Neural Network Hardware Architecture on FPGA