Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication

Abstract

This paper presents a new technique for training networks for low-precision communication. Targeting minimal communication between nodes not only enables the use of emerging spiking neuromorphic platforms, but may additionally streamline processing conventionally. Low-power and embedded neuromorphic processors potentially offer dramatic performance-per-Watt improvements over traditional von Neumann processors, however programming these brain-inspired platforms generally requires platform-specific expertise which limits their applicability. To date, the majority of artificial neural networks have not operated using discrete spike-like communication. We present a method for training deep spiking neural networks using an iterative modification of the backpropagation optimization algorithm. This method, which we call Whetstone, effectively and reliably configures a network for a spiking hardware target with little, if any, loss in performance. Whetstone networks use single time step binary communication and do not require a rate code or other spike-based coding scheme, thus producing networks comparable in timing and size to conventional ANNs, albeit with binarized communication. We demonstrate Whetstone on a number of image classification networks, describing how the sharpening process interacts with different training optimizers and changes the distribution of activity within the network. We further note that Whetstone is compatible with several non-classification neural network applications, such as autoencoders and semantic segmentation. Whetstone is widely extendable and currently implemented using custom activation functions within the Keras wrapper to the popular TensorFlow machine learning framework.

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Binarized Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Learning to Segment Object Candidates
• A Survey of Neuromorphic Computing and Neural Networks in Hardware
• NVIDIA Tensor Core Programmability, Performance & Precision
• Spiking Deep Networks with LIF Neurons
• Training Spiking Deep Networks for Neuromorphic Hardware

Cited by in corpus (12)

• Constructing Accurate and Efficient Deep Spiking Neural Networks with Double-threshold and Augmented Schemes
• Neuromorphic scaling advantages for energy-efficient random walk computation
• The Backpropagation Algorithm Implemented on Spiking Neuromorphic Hardware
• Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks
• Synaptic Learning with Augmented Spikes
• Integration of Neuromorphic AI in Event-Driven Distributed Digitized Systems: Concepts and Research Directions
• Applications of Spiking Neural Networks in Visual Place Recognition
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks
• SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference
• Towards Scalable, Efficient and Accurate Deep Spiking Neural Networks with Backward Residual Connections, Stochastic Softmax and Hybridization
• Analytical aspects of non-differentiable neural networks