BinaryConnect: Training Deep Neural Networks with binary weights during propagations
arXiv:1511.00363
Abstract
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.
arXiv commentAccepted at NIPS 2015, 9 pages, 3 figures
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Expectation Propagation for approximate Bayesian inference
- Going Deeper with Convolutions
- Theano: new features and speed improvements
- Deeply-Supervised Nets
- Deep Learning using Linear Support Vector Machines
- Spatially-sparse convolutional neural networks
- Pylearn2: a machine learning research library
- Rounding Methods for Neural Networks with Low Resolution Synaptic Weights
- Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation
Cited by in corpus (281)
- Knowledge Distillation: A Survey
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- AutoML: A Survey of the State-of-the-Art
- Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Deep Face Recognition: A Survey
- Mixed Precision Training
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
- Ternary Weight Networks
- Trained Ternary Quantization
- Group Sparse Regularization for Deep Neural Networks
- Optimization Problems for Machine Learning: A Survey
- Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
- Bayesian Compression for Deep Learning
- Training and Inference with Integers in Deep Neural Networks
- NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
- Hardware-oriented Approximation of Convolutional Neural Networks
- Model compression via distillation and quantization
- Hierarchical Multiscale Recurrent Neural Networks
- A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
- Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
- First-spike based visual categorization using reward-modulated STDP
- Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
- A Survey on Methods and Theories of Quantized Neural Networks
- WRPN: Wide Reduced-Precision Networks
- Online Batch Selection for Faster Training of Neural Networks
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- Neural Networks with Few Multiplications
- Distilled Siamese Networks for Visual Tracking
- Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
- An Entropy-based Pruning Method for CNN Compression
- Resiliency of Deep Neural Networks under Quantization
- Dynamic Network Surgery for Efficient DNNs
- Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
- Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
- XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Recurrent Neural Networks: An Embedded Computing Perspective
- Mixed Precision Training of Convolutional Neural Networks using Integer Operations
- Mixed-precision deep learning based on computational memory
- Compressing Neural Networks using the Variational Information Bottleneck
- Variable Rate Image Compression with Recurrent Neural Networks
- Knowledge Distillation in Deep Learning and its Applications
- Training with Quantization Noise for Extreme Model Compression
- Compressing Recurrent Neural Network with Tensor Train
- Accelerating CNN inference on FPGAs: A Survey
- Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
- The Power of Sparsity in Convolutional Neural Networks
- Loss-aware Weight Quantization of Deep Networks
- BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning
- NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
- Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
- Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
- Embedded Binarized Neural Networks
- Deep neural networks are robust to weight binarization and other non-linear distortions
- Secure Evaluation of Quantized Neural Networks
- Distributed Deep Neural Networks over the Cloud, the Edge and End Devices
- AddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
- Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
- Deep Learning with Low Precision by Half-wave Gaussian Quantization
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks
- MaskConnect: Connectivity Learning by Gradient Descent
- TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
- Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
- TBD: Benchmarking and Analyzing Deep Neural Network Training
- Automated Pruning for Deep Neural Network Compression
- Deep Spiking Networks
- Automated Verification of Neural Networks: Advances, Challenges and Perspectives
- Low-complexity Approximate Convolutional Neural Networks
- DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
- An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
- HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
- Efficient Sparse-Winograd Convolutional Neural Networks
- BinaryBERT: Pushing the Limit of BERT Quantization
- A Survey on Symbolic Knowledge Distillation of Large Language Models
- Delta Networks for Optimized Recurrent Network Computation
- Hierarchical binary CNNs for landmark localization with limited resources
- RANC: Reconfigurable Architecture for Neuromorphic Computing
- Quantization of Fully Convolutional Networks for Accurate Biomedical Image Segmentation
- Compact Deep Convolutional Neural Networks With Coarse Pruning
- Progressive Weight Pruning of Deep Neural Networks using ADMM
- More is Less: A More Complicated Network with Less Inference Complexity
- On-device Training: A First Overview on Existing Systems
- An Improvement of Data Classification Using Random Multimodel Deep Learning (RMDL)
- Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
- A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
- Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
- ExPAN(N)D: Exploring Posits for Efficient Artificial Neural Network Design in FPGA-based Systems
- Controlling Information Capacity of Binary Neural Network
- Energy Efficient Learning with Low Resolution Stochastic Domain Wall Synapse Based Deep Neural Networks
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
- Rethinking Weight Decay For Efficient Neural Network Pruning
- Learning Sparse Low-Precision Neural Networks With Learnable Regularization
- Robustness of Neural Networks against Storage Media Errors
- Communication-Efficient Edge AI: Algorithms and Systems
- Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
- Model compression as constrained optimization, with application to neural nets. Part I: general framework
- How fine can fine-tuning be? Learning efficient language models
- Training DNNs with Hybrid Block Floating Point
- Connectivity Learning in Multi-Branch Networks
- Computing Systems for Autonomous Driving: State-of-the-Art and Challenges
- Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
- Quantization for Rapid Deployment of Deep Neural Networks
- Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
- Model compression as constrained optimization, with application to neural nets. Part II: quantization
- Programming and Training Rate-Independent Chemical Reaction Networks
- AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
- Towards Effective Low-bitwidth Convolutional Neural Networks
- Optimally Scheduling CNN Convolutions for Efficient Memory Access
- Towards Unified INT8 Training for Convolutional Neural Network
- Deep Learning for Real-Time Crime Forecasting and its Ternarization
- Q-CapsNets: A Specialized Framework for Quantizing Capsule Networks
- Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
- Degree-Quant: Quantization-Aware Training for Graph Neural Networks
- Regularizing Activation Distribution for Training Binarized Deep Networks
- Computer Vision Model Compression Techniques for Embedded Systems: A Survey
- Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
- Differentiable Model Compression via Pseudo Quantization Noise
- Performance Guaranteed Network Acceleration via High-Order Residual Quantization
- Binarized Neural Networks on the ImageNet Classification Task
- Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization
- Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
- Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
- Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
- Incremental Binarization On Recurrent Neural Networks For Single-Channel Source Separation
- Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
- Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
- Iteratively Training Look-Up Tables for Network Quantization
- Taxonomy and Benchmarking of Precision-Scalable MAC Arrays Under Enhanced DNN Dataflow Representation
- Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits
- Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
- Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
- Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
- Exploiting Errors for Efficiency: A Survey from Circuits to Algorithms
- An Encoding Framework for Binarized Images using HyperDimensional Computing
- Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
- BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
- Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
- SEP-Nets: Small and Effective Pattern Networks
- In-situ Stochastic Training of MTJ Crossbar based Neural Networks
- Searching for Winograd-aware Quantized Networks
- On Periodic Functions as Regularizers for Quantization of Neural Networks
- OpenEI: An Open Framework for Edge Intelligence
- DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics
- LCNN: Lookup-based Convolutional Neural Network
- Fix your classifier: the marginal value of training the last weight layer
- Towards Evolutional Compression
- Adjustable Bounded Rectifiers: Towards Deep Binary Representations
- Temporally Efficient Deep Learning with Spikes
- Efficient Stochastic Inference of Bitwise Deep Neural Networks
- BiFSMN: Binary Neural Network for Keyword Spotting
- Low-Precision Batch-Normalized Activations
- Mixed-precision training of deep neural networks using computational memory
- BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet
- Sparse Weight Activation Training
- Joint Pruning & Quantization for Extremely Sparse Neural Networks
- HEMP: High-order Entropy Minimization for neural network comPression
- Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks
- Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy
- Deep Feature Flow for Video Recognition
- Espresso: Efficient Forward Propagation for BCNNs
- SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
- Learning Versatile Convolution Filters for Efficient Visual Recognition
- Deep causal representation learning for unsupervised domain adaptation
- Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks
- Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
- Accelerating Deterministic and Stochastic Binarized Neural Networks on FPGAs Using OpenCL
- Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
- Tag Prediction at Flickr: a View from the Darkroom
- Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
- Recursive Binary Neural Network Learning Model with 2.28b/Weight Storage Requirement
- Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks
- Deep Dose Plugin Towards Real-time Monte Carlo Dose Calculation Through a Deep Learning based Denoising Algorithm
- GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
- Taming Binarized Neural Networks and Mixed-Integer Programs
- On the efficient representation and execution of deep acoustic models
- At-Scale Evaluation of Weight Clustering to Enable Energy-Efficient Object Detection
- DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization
- A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
- A Simple Method to Reduce Off-chip Memory Accesses on Convolutional Neural Networks
- SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
- Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks
- BitNet: Bit-Regularized Deep Neural Networks
- CNN Acceleration by Low-rank Approximation with Quantized Factors
- Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices
- Efficient Hardware Realization of Convolutional Neural Networks using Intra-Kernel Regular Pruning
- A GPU-Outperforming FPGA Accelerator Architecture for Binary Convolutional Neural Networks
- Compressing complex convolutional neural network based on an improved deep compression algorithm
- Quantized Non-Volatile Nanomagnetic Synapse based Autoencoder for Efficient Unsupervised Network Anomaly Detection
- Differentiable Sparsification for Deep Neural Networks
- Approximations in Deep Learning
- Detecting Dead Weights and Units in Neural Networks
- Binarized Convolutional Neural Networks with Separable Filters for Efficient Hardware Acceleration
- Matrix and tensor decompositions for training binary neural networks
- Training for 'Unstable' CNN Accelerator:A Case Study on FPGA
- Some Remarks on Replicated Simulated Annealing
- Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
- Convolutional Neural Network Quantization using Generalized Gamma Distribution
- Training Deep Neural Network in Limited Precision
- Privacy-Preserving Visual Learning Using Doubly Permuted Homomorphic Encryption
- Simplified Stochastic Feedforward Neural Networks
- Knowledge distillation for optimization of quantized deep neural networks
- Binary Complex Neural Network Acceleration on FPGA
- A High-Performance Adaptive Quantization Approach for Edge CNN Applications
- Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimization
- Compressing LSTM Networks by Matrix Product Operators
- Digital Neuron: A Hardware Inference Accelerator for Convolutional Deep Neural Networks
- Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network
- Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
- Low Precision Policy Distillation with Application to Low-Power, Real-time Sensation-Cognition-Action Loop with Neuromorphic Computing
- Learning Recurrent Binary/Ternary Weights
- A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
- n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
- Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
- Hierarchical compositional feature learning
- Toward Computation and Memory Efficient Neural Network Acoustic Models with Binary Weights and Activations
- Local Binary Pattern Networks
- BCNN: Binary Complex Neural Network
- TinBiNN: Tiny Binarized Neural Network Overlay in about 5,000 4-LUTs and 5mW
- Integer-Only Neural Network Quantization Scheme Based on Shift-Batch-Normalization
- Binarized Knowledge Graph Embeddings
- Training of Quantized Deep Neural Networks using a Magnetic Tunnel Junction-Based Synapse
- Fixed-point optimization of deep neural networks with adaptive step size retraining
- Leveraging Structured Pruning of Convolutional Neural Networks
- AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
- Binary-decomposed DCNN for accelerating computation and compressing model without retraining
- Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
- Recent Advances in Convolutional Neural Network Acceleration
- Learning Multimodal Fixed-Point Weights using Gradient Descent
- Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
- Quantization Mimic: Towards Very Tiny CNN for Object Detection
- Finding Everything within Random Binary Networks
- Understanding the Energy and Precision Requirements for Online Learning
- NEAT: A Framework for Automated Exploration of Floating Point Approximations
- A New Training Framework for Deep Neural Network
- Quantized neural network design under weight capacity constraint
- "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
- Convolutional neural networks compression with low rank and sparse tensor decompositions
- Automated Deep Abstractions for Stochastic Chemical Reaction Networks
- A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
- A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
- Energy Consumption Analysis of pruned Semantic Segmentation Networks on an Embedded GPU
- On Transformations in Stochastic Gradient MCMC
- Composite Binary Decomposition Networks
- Learning Instance-wise Sparsity for Accelerating Deep Models
- Efficient and Robust Machine Learning for Real-World Systems
- Classification Accuracy Improvement for Neuromorphic Computing Systems with One-level Precision Synapses
- Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness
- Markov chain Hebbian learning algorithm with ternary synaptic units
- Penetrating the Fog: the Path to Efficient CNN Models
- ENOS: Energy-Aware Network Operator Search for Hybrid Digital and Compute-in-Memory DNN Accelerators
- Adaptive Binary-Ternary Quantization
- Selective Deep Convolutional Neural Network for Low Cost Distorted Image Classification
- Learning to Skip Ineffectual Recurrent Computations in LSTMs
- Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
- Binarized Convolutional Neural Networks for Efficient Inference on GPUs
- Fixed-point Factorized Networks
- Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
- ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition
- Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning
- A Greedy Algorithm to Cluster Specialists
- Collaborative Deep Learning for Speech Enhancement: A Run-Time Model Selection Method Using Autoencoders
- Reliable Identification of Redundant Kernels for Convolutional Neural Network Compression
- A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)
- Bayesian Optimized 1-Bit CNNs
- ATP-Net: An Attention-based Ternary Projection Network For Compressed Sensing
- Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagation
- Robust Implicit Backpropagation
- Recurrent Residual Module for Fast Inference in Videos
- Cyber-Secure Wearable, Ultra-Low-Power Networked Sensors for Soldier Health Monitoring: Insights from the CAPSARII project
- An Overview of Datatype Quantization Techniques for Convolutional Neural Networks
- RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
- Homomorphic Parameter Compression for Distributed Deep Learning Training
- : Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
- Neural Network Activation Quantization with Bitwise Information Bottlenecks
- Maximin Optimization for Binary Regression
- Training compact deep learning models for video classification using circulant matrices