Mapping high-performance RNNs to in-memory neuromorphic chips

Abstract

The increasing need for compact and low-power computing solutions for machine learning applications has triggered significant interest in energy-efficient neuromorphic systems. However, most of these architectures rely on spiking neural networks, which typically perform poorly compared to their non-spiking counterparts in terms of accuracy. In this paper, we propose a new adaptive spiking neuron model that can be abstracted as a low-pass filter. This abstraction enables faster and better training of spiking networks using back-propagation, without simulating spikes. We show that this model dramatically improves the inference performance of a recurrent neural network and validate it with three complex spatio-temporal learning tasks: the temporal addition task, the temporal copying task, and a spoken-phrase recognition task. We estimate at least 500x higher energy-efficiency using our models on compatible neuromorphic chips in comparison to Cortex-M4, a popular embedded microprocessor.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Character-Aware Neural Language Models
• A scalable multi-core architecture with heterogeneous memory structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs)
• Neuromorphic electronic circuits for building autonomous cognitive systems
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Origami: A 803 GOp/s/W Convolutional Network Accelerator
• A neuromorphic systems approach to in-memory computing with non-ideal memristive devices: From mitigation to exploitation
• Overcoming the vanishing gradient problem in plain recurrent networks
• Recurrent Orthogonal Networks and Long-Memory Tasks
• Enabling the Adoption of Processing-in-Memory: Challenges, Mechanisms, Future Research Directions
• An ultra-low-power sigma-delta neuron circuit