Training large-scale ANNs on simulated resistive crossbar arrays

Abstract

Accelerating training of artificial neural networks (ANN) with analog resistive crossbar arrays is a promising idea. While the concept has been verified on very small ANNs and toy data sets (such as MNIST), more realistically sized ANNs and datasets have not yet been tackled. However, it is to be expected that device materials and hardware design constraints, such as noisy computations, finite number of resistive states of the device materials, saturating weight and activation ranges, and limited precision of analog-to-digital converters, will cause significant challenges to the successful training of state-of-the-art ANNs. By using analog hardware aware ANN training simulations, we here explore a number of simple algorithmic compensatory measures to cope with analog noise and limited weight and output ranges and resolutions, that dramatically improve the simulated training performances on RPU arrays on intermediately to large-scale ANNs.

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Training LSTM Networks with Resistive Cross-Point Devices
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Efficient ConvNets for Analog Arrays

Cited by in corpus (4)

• Using the IBM Analog In-Memory Hardware Acceleration Kit for Neural Network Training and Inference
• Fast offset corrected in-memory training
• Study of Resistive Switching Dynamics and Memory States Equilibria in Analog Filamentary Conductive-Metal-Oxide/HfOx ReRAM via Compact Modeling
• Digital-analog concept for superconducting perceptron-like neural networks