Deep neural networks are robust to weight binarization and other non-linear distortions

Abstract

Recent results show that deep neural networks achieve excellent performance even when, during training, weights are quantized and projected to a binary representation. Here, we show that this is just the tip of the iceberg: these same networks, during testing, also exhibit a remarkable robustness to distortions beyond quantization, including additive and multiplicative noise, and a class of non-linear projections where binarization is just a special case. To quantify this robustness, we show that one such network achieves 11% test error on CIFAR-10 even with 0.68 effective bits per weight. Furthermore, we find that a common training heuristic--namely, projecting quantized weights during backpropagation--can be altered (or even removed) and networks still achieve a base level of robustness during testing. Specifically, training with weight projections other than quantization also works, as does simply clipping the weights, both of which have never been reported before. We confirm our results for CIFAR-10 and ImageNet datasets. Finally, drawing from these ideas, we propose a stochastic projection rule that leads to a new state of the art network with 7.64% test error on CIFAR-10 using no data augmentation.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Pylearn2: a machine learning research library
• Memory Bounded Deep Convolutional Networks

Cited by in corpus (19)

• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Bayesian Compression for Deep Learning
• A Survey on Methods and Theories of Quantized Neural Networks
• Mixed-precision deep learning based on computational memory
• Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• Hierarchical binary CNNs for landmark localization with limited resources
• FQ-Conv: Fully Quantized Convolution for Efficient and Accurate Inference
• Stein Neural Sampler
• Low-Precision Batch-Normalized Activations
• Mixed-precision training of deep neural networks using computational memory
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Training Generative Adversarial Networks via Primal-Dual Subgradient Methods: A Lagrangian Perspective on GAN
• BitNet: Bit-Regularized Deep Neural Networks
• Mitigating Imperfections in Mixed-Signal Neuromorphic Circuits
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• R2F: A Remote Retraining Framework for AIoT Processors with Computing Errors