Streaming Batch Eigenupdates for Hardware Neuromorphic Networks

Abstract

Neuromorphic networks based on nanodevices, such as metal oxide memristors, phase change memories, and flash memory cells, have generated considerable interest for their increased energy efficiency and density in comparison to graphics processing units (GPUs) and central processing units (CPUs). Though immense acceleration of the training process can be achieved by leveraging the fact that the time complexity of training does not scale with the network size, it is limited by the space complexity of stochastic gradient descent, which grows quadratically. The main objective of this work is to reduce this space complexity by using low-rank approximations of stochastic gradient descent. This low spatial complexity combined with streaming methods allows for significant reductions in memory and compute overhead, opening the doors for improvements in area, time and energy efficiency of training. We refer to this algorithm and architecture to implement it as the streaming batch eigenupdate (SBE) approach.

arXiv comment13 pages, 5 figures

References in corpus (11)

• Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors
• Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices
• The Noisy Power Method: A Meta Algorithm with Applications
• Randomized Block Krylov Methods for Stronger and Faster Approximate Singular Value Decomposition
• Memory Limited, Streaming PCA
• Analog CMOS-based Resistive Processing Unit for Deep Neural Network Training
• Training LSTM Networks with Resistive Cross-Point Devices
• The Fast Convergence of Incremental PCA
• Streaming Batch Eigenupdates for Hardware Neuromorphic Networks
• History PCA: A New Algorithm for Streaming PCA
• PCA of high dimensional random walks with comparison to neural network training

Cited by in corpus (5)

• Training End-to-End Analog Neural Networks with Equilibrium Propagation
• Streaming Batch Eigenupdates for Hardware Neuromorphic Networks
• Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition
• Memory-efficient training with streaming dimensionality reduction
• Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions