Memorized Sparse Backpropagation

Abstract

Neural network learning is usually time-consuming since backpropagation needs to compute full gradients and backpropagate them across multiple layers. Despite its success of existing works in accelerating propagation through sparseness, the relevant theoretical characteristics remain under-researched and empirical studies found that they suffer from the loss of information contained in unpropagated gradients. To tackle these problems, this paper presents a unified sparse backpropagation framework and provides a detailed analysis of its theoretical characteristics. Analysis reveals that when applied to a multilayer perceptron, our framework essentially performs gradient descent using an estimated gradient similar enough to the true gradient, resulting in convergence in probability under certain conditions. Furthermore, a simple yet effective algorithm named memorized sparse backpropagation (MSBP) is proposed to remedy the problem of information loss by storing unpropagated gradients in memory for learning in the next steps. Experimental results demonstrate that the proposed MSBP is effective to alleviate the information loss in traditional sparse backpropagation while achieving comparable acceleration.

arXiv commentAccepted to Neurocomputing

References in corpus (13)

• ADADELTA: An Adaptive Learning Rate Method
• Natural Language Processing (almost) from Scratch
• On the difficulty of training Recurrent Neural Networks
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Non-convex Optimization for Machine Learning
• Local SGD Converges Fast and Communicates Little
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• A Unified Tagging Solution: Bidirectional LSTM Recurrent Neural Network with Word Embedding
• Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation
• Minimal Effort Back Propagation for Convolutional Neural Networks
• Local Propagation in Constraint-based Neural Network
• PJAIT Systems for the IWSLT 2015 Evaluation Campaign Enhanced by Comparable Corpora

Cited by in corpus (2)

• Towards Accurate Post-Training Quantization for Vision Transformer
• Accelerating CNN Training by Pruning Activation Gradients