BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Abstract

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

Accepted at NIPS 2015, 9 pages, 3 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Expectation Propagation for approximate Bayesian inference
• Going Deeper with Convolutions
• Theano: new features and speed improvements
• Deeply-Supervised Nets
• Deep Learning using Linear Support Vector Machines
• Spatially-sparse convolutional neural networks
• Pylearn2: a machine learning research library
• Rounding Methods for Neural Networks with Low Resolution Synaptic Weights
• Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation

Cited by in corpus (280)

• Knowledge Distillation: A Survey
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• AutoML: A Survey of the State-of-the-Art
• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Face Recognition: A Survey
• Mixed Precision Training
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Ternary Weight Networks
• Trained Ternary Quantization
• Group Sparse Regularization for Deep Neural Networks
• Optimization Problems for Machine Learning: A Survey
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• Bayesian Compression for Deep Learning
• Training and Inference with Integers in Deep Neural Networks
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Hardware-oriented Approximation of Convolutional Neural Networks
• Model compression via distillation and quantization
• Hierarchical Multiscale Recurrent Neural Networks
• A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• First-spike based visual categorization using reward-modulated STDP
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A Survey on Methods and Theories of Quantized Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Online Batch Selection for Faster Training of Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Distilled Siamese Networks for Visual Tracking
• Neural Networks with Few Multiplications
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• An Entropy-based Pruning Method for CNN Compression
• Resiliency of Deep Neural Networks under Quantization
• Dynamic Network Surgery for Efficient DNNs
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Recurrent Neural Networks: An Embedded Computing Perspective
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Compressing Neural Networks using the Variational Information Bottleneck
• Mixed-precision deep learning based on computational memory
• Knowledge Distillation in Deep Learning and its Applications
• Variable Rate Image Compression with Recurrent Neural Networks
• Training with Quantization Noise for Extreme Model Compression
• Compressing Recurrent Neural Network with Tensor Train
• Accelerating CNN inference on FPGAs: A Survey
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• The Power of Sparsity in Convolutional Neural Networks
• Loss-aware Weight Quantization of Deep Networks
• BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning
• NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
• Embedded Binarized Neural Networks
• Secure Evaluation of Quantized Neural Networks
• Deep neural networks are robust to weight binarization and other non-linear distortions
• Distributed Deep Neural Networks over the Cloud, the Edge and End Devices
• AddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• MaskConnect: Connectivity Learning by Gradient Descent
• TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
• Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• Automated Pruning for Deep Neural Network Compression
• Deep Spiking Networks
• Automated Verification of Neural Networks: Advances, Challenges and Perspectives
• DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
• Low-complexity Approximate Convolutional Neural Networks
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• Efficient Sparse-Winograd Convolutional Neural Networks
• BinaryBERT: Pushing the Limit of BERT Quantization
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Delta Networks for Optimized Recurrent Network Computation
• Hierarchical binary CNNs for landmark localization with limited resources
• RANC: Reconfigurable Architecture for Neuromorphic Computing
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• More is Less: A More Complicated Network with Less Inference Complexity
• Quantization of Fully Convolutional Networks for Accurate Biomedical Image Segmentation
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• An Improvement of Data Classification Using Random Multimodel Deep Learning (RMDL)
• On-device Training: A First Overview on Existing Systems
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
• ExPAN(N)D: Exploring Posits for Efficient Artificial Neural Network Design in FPGA-based Systems
• Controlling Information Capacity of Binary Neural Network
• Energy Efficient Learning with Low Resolution Stochastic Domain Wall Synapse Based Deep Neural Networks
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• How fine can fine-tuning be? Learning efficient language models
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Training DNNs with Hybrid Block Floating Point
• Robustness of Neural Networks against Storage Media Errors
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• Communication-Efficient Edge AI: Algorithms and Systems
• Connectivity Learning in Multi-Branch Networks
• Computing Systems for Autonomous Driving: State-of-the-Art and Challenges
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• Quantization for Rapid Deployment of Deep Neural Networks
• Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
• Programming and Training Rate-Independent Chemical Reaction Networks
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• Towards Effective Low-bitwidth Convolutional Neural Networks
• Optimally Scheduling CNN Convolutions for Efficient Memory Access
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• Q-CapsNets: A Specialized Framework for Quantizing Capsule Networks
• Deep Learning for Real-Time Crime Forecasting and its Ternarization
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• Towards Unified INT8 Training for Convolutional Neural Network
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
• Differentiable Model Compression via Pseudo Quantization Noise
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• Binarized Neural Networks on the ImageNet Classification Task
• Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• Incremental Binarization On Recurrent Neural Networks For Single-Channel Source Separation
• Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
• Taxonomy and Benchmarking of Precision-Scalable MAC Arrays Under Enhanced DNN Dataflow Representation
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Iteratively Training Look-Up Tables for Network Quantization
• Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
• Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits
• Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
• Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
• Exploiting Errors for Efficiency: A Survey from Circuits to Algorithms
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• An Encoding Framework for Binarized Images using HyperDimensional Computing
• Searching for Winograd-aware Quantized Networks
• In-situ Stochastic Training of MTJ Crossbar based Neural Networks
• SEP-Nets: Small and Effective Pattern Networks
• OpenEI: An Open Framework for Edge Intelligence
• DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics
• On Periodic Functions as Regularizers for Quantization of Neural Networks
• LCNN: Lookup-based Convolutional Neural Network
• Towards Evolutional Compression
• Fix your classifier: the marginal value of training the last weight layer
• Low-Precision Batch-Normalized Activations
• Adjustable Bounded Rectifiers: Towards Deep Binary Representations
• Efficient Stochastic Inference of Bitwise Deep Neural Networks
• BiFSMN: Binary Neural Network for Keyword Spotting
• Mixed-precision training of deep neural networks using computational memory
• Temporally Efficient Deep Learning with Spikes
• Joint Pruning & Quantization for Extremely Sparse Neural Networks
• BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet
• Sparse Weight Activation Training
• Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks
• Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy
• HEMP: High-order Entropy Minimization for neural network comPression
• Espresso: Efficient Forward Propagation for BCNNs
• Deep Feature Flow for Video Recognition
• SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks
• Deep causal representation learning for unsupervised domain adaptation
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Accelerating Deterministic and Stochastic Binarized Neural Networks on FPGAs Using OpenCL
• Recursive Binary Neural Network Learning Model with 2.28b/Weight Storage Requirement
• Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks
• At-Scale Evaluation of Weight Clustering to Enable Energy-Efficient Object Detection
• Taming Binarized Neural Networks and Mixed-Integer Programs
• Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• Tag Prediction at Flickr: a View from the Darkroom
• GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• Deep Dose Plugin Towards Real-time Monte Carlo Dose Calculation Through a Deep Learning based Denoising Algorithm
• On the efficient representation and execution of deep acoustic models
• A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices
• Efficient Hardware Realization of Convolutional Neural Networks using Intra-Kernel Regular Pruning
• Compressing complex convolutional neural network based on an improved deep compression algorithm
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• Differentiable Sparsification for Deep Neural Networks
• BitNet: Bit-Regularized Deep Neural Networks
• A Simple Method to Reduce Off-chip Memory Accesses on Convolutional Neural Networks
• Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks
• A GPU-Outperforming FPGA Accelerator Architecture for Binary Convolutional Neural Networks
• DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization
• Quantized Non-Volatile Nanomagnetic Synapse based Autoencoder for Efficient Unsupervised Network Anomaly Detection
• Simplified Stochastic Feedforward Neural Networks
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• Approximations in Deep Learning
• Knowledge distillation for optimization of quantized deep neural networks
• Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimization
• Some Remarks on Replicated Simulated Annealing
• Matrix and tensor decompositions for training binary neural networks
• Training for 'Unstable' CNN Accelerator:A Case Study on FPGA
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• Convolutional Neural Network Quantization using Generalized Gamma Distribution
• Training Deep Neural Network in Limited Precision
• Detecting Dead Weights and Units in Neural Networks
• Binary Complex Neural Network Acceleration on FPGA
• Binarized Convolutional Neural Networks with Separable Filters for Efficient Hardware Acceleration
• Privacy-Preserving Visual Learning Using Doubly Permuted Homomorphic Encryption
• Hierarchical compositional feature learning
• Learning Recurrent Binary/Ternary Weights
• Low Precision Policy Distillation with Application to Low-Power, Real-time Sensation-Cognition-Action Loop with Neuromorphic Computing
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
• Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network
• Compressing LSTM Networks by Matrix Product Operators
• n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
• Digital Neuron: A Hardware Inference Accelerator for Convolutional Deep Neural Networks
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Toward Computation and Memory Efficient Neural Network Acoustic Models with Binary Weights and Activations
• Leveraging Structured Pruning of Convolutional Neural Networks
• TinBiNN: Tiny Binarized Neural Network Overlay in about 5,000 4-LUTs and 5mW
• Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
• Local Binary Pattern Networks
• BCNN: Binary Complex Neural Network
• Binarized Knowledge Graph Embeddings
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Training of Quantized Deep Neural Networks using a Magnetic Tunnel Junction-Based Synapse
• Fixed-point optimization of deep neural networks with adaptive step size retraining
• Learning Multimodal Fixed-Point Weights using Gradient Descent
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Binary-decomposed DCNN for accelerating computation and compressing model without retraining
• Integer-Only Neural Network Quantization Scheme Based on Shift-Batch-Normalization
• Recent Advances in Convolutional Neural Network Acceleration
• Efficient and Robust Machine Learning for Real-World Systems
• Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• Automated Deep Abstractions for Stochastic Chemical Reaction Networks
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• Composite Binary Decomposition Networks
• NEAT: A Framework for Automated Exploration of Floating Point Approximations
• Learning Instance-wise Sparsity for Accelerating Deep Models
• On Transformations in Stochastic Gradient MCMC
• Energy Consumption Analysis of pruned Semantic Segmentation Networks on an Embedded GPU
• Understanding the Energy and Precision Requirements for Online Learning
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Classification Accuracy Improvement for Neuromorphic Computing Systems with One-level Precision Synapses
• Finding Everything within Random Binary Networks
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• A New Training Framework for Deep Neural Network
• Quantized neural network design under weight capacity constraint
• A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
• Markov chain Hebbian learning algorithm with ternary synaptic units
• Homomorphic Parameter Compression for Distributed Deep Learning Training
• ENOS: Energy-Aware Network Operator Search for Hybrid Digital and Compute-in-Memory DNN Accelerators
• Maximin Optimization for Binary Regression
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• ATP-Net: An Attention-based Ternary Projection Network For Compressed Sensing
• Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagation
• $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
• Adaptive Binary-Ternary Quantization
• Bayesian Optimized 1-Bit CNNs
• Reliable Identification of Redundant Kernels for Convolutional Neural Network Compression
• Learning to Skip Ineffectual Recurrent Computations in LSTMs
• Penetrating the Fog: the Path to Efficient CNN Models
• Training compact deep learning models for video classification using circulant matrices
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• An Overview of Datatype Quantization Techniques for Convolutional Neural Networks
• Robust Implicit Backpropagation
• A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)
• Binarized Convolutional Neural Networks for Efficient Inference on GPUs
• ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition
• Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
• Selective Deep Convolutional Neural Network for Low Cost Distorted Image Classification
• Collaborative Deep Learning for Speech Enhancement: A Run-Time Model Selection Method Using Autoencoders
• Recurrent Residual Module for Fast Inference in Videos
• Neural Network Activation Quantization with Bitwise Information Bottlenecks
• Fixed-point Factorized Networks
• Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning
• A Greedy Algorithm to Cluster Specialists