A Gradient-Interleaved Scheduler for Energy-Efficient Backpropagation for Training Neural Networks

Abstract

This paper addresses design of accelerators using systolic architectures for training of neural networks using a novel gradient interleaving approach. Training the neural network involves backpropagation of error and computation of gradients with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients to the same configurable systolic array. This results in reuse of the variables from one computation to the other and eliminates unnecessary memory accesses. The proposed approach leads to 1.4 - 2.2 times savings in terms of number of cycles and $1.9\times$ savings in terms of memory accesses. Thus, the proposed accelerator reduces latency and energy consumption.

arXiv commentProc. 2020 IEEE International Symposium on Circuits and Systems (ISCAS)

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SCALE-Sim: Systolic CNN Accelerator Simulator
• EIE: Efficient Inference Engine on Compressed Deep Neural Network

Cited by in corpus (3)

• LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling
• SCV-GNN: Sparse Compressed Vector-based Graph Neural Network Aggregation
• LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks