Probabilistic Binary Neural Networks

Abstract

Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called BLRNet. By embracing stochasticity during training, we circumvent the need to approximate the gradient of non-differentiable functions such as sign(), while still obtaining a fully Binary Neural Network at test time. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since all operations in a layer of the BLRNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time. We evaluate the BLRNet on multiple standardized benchmarks.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• Wide Residual Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Binarized Neural Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Towards Accurate Binary Convolutional Neural Network
• Bayesian Compression for Deep Learning
• Training wide residual networks for deployment using a single bit for each weight

Cited by in corpus (19)

• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• Resource-Efficient Neural Networks for Embedded Systems
• Training Binary Neural Networks using the Bayesian Learning Rule
• Relaxed Quantization for Discretized Neural Networks
• Bit Efficient Quantization for Deep Neural Networks
• Generative Zero-shot Network Quantization
• A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
• Critical initialisation in continuous approximations of binary neural networks
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• Analytically Tractable Inference in Deep Neural Networks
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• Quantum Deformed Neural Networks
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• SiMaN: Sign-to-Magnitude Network Binarization
• ENOS: Energy-Aware Network Operator Search for Hybrid Digital and Compute-in-Memory DNN Accelerators
• Demystifying and Generalizing BinaryConnect
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• Sparsity-Control Ternary Weight Networks