Bitwise Neural Networks

Abstract

Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.

This paper was presented at the International Conference on Machine Learning (ICML) Workshop on Resource-Efficient Machine Learning, Lille, France, Jul. 6-11, 2015

Cited by in corpus (55)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Binary Neural Networks: A Survey
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• A Survey on Methods and Theories of Quantized Neural Networks
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• A neural network memory prefetcher using semantic locality
• TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
• Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks
• Low-complexity Approximate Convolutional Neural Networks
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• In-network Neural Networks
• FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
• QUOTIENT: Two-Party Secure Neural Network Training and Prediction
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• AutoQ: Automated Kernel-Wise Neural Network Quantization
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Incremental Binarization On Recurrent Neural Networks For Single-Channel Source Separation
• Energy Efficient Hadamard Neural Networks
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding
• LCNN: Lookup-based Convolutional Neural Network
• An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions
• Training Bit Fully Convolutional Network for Fast Semantic Segmentation
• Scaling Binarized Neural Networks on Reconfigurable Logic
• On the efficient representation and execution of deep acoustic models
• Accurate and Efficient Hyperbolic Tangent Activation Function on FPGA using the DCT Interpolation Filter
• Verification of Binarized Neural Networks via Inter-Neuron Factoring
• A GPU-Outperforming FPGA Accelerator Architecture for Binary Convolutional Neural Networks
• Binarized Convolutional Neural Networks with Separable Filters for Efficient Hardware Acceleration
• Do We Need Fully Connected Output Layers in Convolutional Networks?
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• On Psychoacoustically Weighted Cost Functions Towards Resource-Efficient Deep Neural Networks for Speech Denoising
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• The Shape of RemiXXXes to Come: Audio Texture Synthesis with Time-frequency Scattering
• Recent Advances in Convolutional Neural Network Acceleration
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Understanding the Energy and Precision Requirements for Online Learning
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• Classification Accuracy Improvement for Neuromorphic Computing Systems with One-level Precision Synapses
• Sparse Mixture of Local Experts for Efficient Speech Enhancement
• Quantization Loss Re-Learning Method
• Identifying and Exploiting Structures for Reliable Deep Learning
• Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks
• Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks
• Neural Network Activation Quantization with Bitwise Information Bottlenecks
• Smoothed Differential Privacy
• Collaborative Deep Learning for Speech Enhancement: A Run-Time Model Selection Method Using Autoencoders
• Fixed-point Factorized Networks