A Highly Parallel FPGA Implementation of Sparse Neural Network Training

Abstract

We demonstrate an FPGA implementation of a parallel and reconfigurable architecture for sparse neural networks, capable of on-chip training and inference. The network connectivity uses pre-determined, structured sparsity to significantly reduce complexity by lowering memory and computational requirements. The architecture uses a notion of edge-processing, leading to efficient pipelining and parallelization. Moreover, the device can be reconfigured to trade off resource utilization with training time to fit networks and datasets of varying sizes. The combined effects of complexity reduction and easy reconfigurability enable significantly greater exploration of network hyperparameters and structures on-chip. As proof of concept, we show implementation results on an Artix-7 FPGA.

arXiv commentAn abridged version of this work was accepted as a short paper (4 pages) at ReConFig: 2018 International Conference on Reconfigurable Computing and FPGAs. This is the full version of this work

References in corpus (11)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Understanding deep learning requires rethinking generalization
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Predicting Parameters in Deep Learning
• Compressing Neural Networks with the Hashing Trick
• Soft Weight-Sharing for Neural Network Compression
• Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections
• Interleaver Design for Deep Neural Networks
• Accelerating Training of Deep Neural Networks via Sparse Edge Processing
• Characterizing Sparse Connectivity Patterns in Neural Networks

Cited by in corpus (2)

• Pre-Defined Sparse Neural Networks with Hardware Acceleration
• SpRRAM: A Predefined Sparsity Based Memristive Neuromorphic Circuit for Low Power Application