BitNet: Bit-Regularized Deep Neural Networks
arXiv:1708.04788
Abstract
We present a novel optimization strategy for training neural networks which we call "BitNet". The parameters of neural networks are usually unconstrained and have a dynamic range dispersed over all real values. Our key idea is to limit the expressive power of the network by dynamically controlling the range and set of values that the parameters can take. We formulate this idea using a novel end-to-end approach that circumvents the discrete parameter space by optimizing a relaxed continuous and differentiable upper bound of the typical classification loss function. The approach can be interpreted as a regularization inspired by the Minimum Description Length (MDL) principle. For each layer of the network, our approach optimizes real-valued translation and scaling factors and arbitrary precision integer-valued parameters (weights). We empirically compare BitNet to an equivalent unregularized model on the MNIST and CIFAR-10 datasets. We show that BitNet converges faster to a superior quality solution. Additionally, the resulting model has significant savings in memory due to the use of integer-valued parameters.
References in corpus (28)
- Adam: A Method for Stochastic Optimization
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Compressing Deep Convolutional Networks using Vector Quantization
- Trained Ternary Quantization
- Pruning Filters for Efficient ConvNets
- Compressing Neural Networks with the Hashing Trick
- Convolutional neural networks with low-rank regularization
- Convolutional Neural Networks using Logarithmic Data Representation
- Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
- Neural Networks with Few Multiplications
- Resiliency of Deep Neural Networks under Quantization
- Memory Bounded Deep Convolutional Networks
- Deep neural networks are robust to weight binarization and other non-linear distortions
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Towards the Limit of Network Quantization
- Compressing Convolutional Neural Networks
- Two-Bit Networks for Deep Learning on Resource-Constrained Embedded Devices
- Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
- Accelerating Deep Convolutional Networks using low-precision and sparsity
- Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
- Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
- Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks
- Fixed-point optimization of deep neural networks with adaptive step size retraining
- Reducing the Model Order of Deep Neural Networks Using Information Theory