Binarized Neural Networks

Abstract

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time. We conduct two sets of experiments, each based on a different framework, namely Torch7 and Theano, where we train BNNs on MNIST, CIFAR-10 and SVHN, and achieve nearly state-of-the-art results. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which might lead to a great increase in power-efficiency. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available.

This is an obsolete version, up to date version is available here: arXiv:1602.02830

Cited by in corpus (296)

• Deep Face Recognition: A Survey
• Mixed Precision Training
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Object Detection in 20 Years: A Survey
• Binary Neural Networks: A Survey
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Fast-SCNN: Fast Semantic Segmentation Network
• CrypTFlow2: Practical 2-Party Secure Inference
• Learned Step Size Quantization
• Deep Learning for Generic Object Detection: A Survey
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Quantization and Deployment of Deep Neural Networks on Microcontrollers
• ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
• Scalable Methods for 8-bit Training of Neural Networks
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• High-Throughput In-Memory Computing for Binary Deep Neural Networks with Monolithically Integrated RRAM and 90nm CMOS
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Discrimination-aware Network Pruning for Deep Model Compression
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors
• Recurrent Neural Networks: An Embedded Computing Perspective
• Neural Network Model Extraction Attacks in Edge Devices by Hearing Architectural Hints
• Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy
• Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
• Rotated Binary Neural Network
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• A neural network memory prefetcher using semantic locality
• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Automated Pruning for Deep Neural Network Compression
• FPGA-Based CNN Inference Accelerator Synthesized from Multi-Threaded C Software
• Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection
• QKD: Quantization-aware Knowledge Distillation
• BinaryBERT: Pushing the Limit of BERT Quantization
• Practical Lossless Compression with Latent Variables using Bits Back Coding
• Back to Simplicity: How to Train Accurate BNNs from Scratch?
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Probabilistic Binary Neural Networks
• Deep Ranking Model by Large Adaptive Margin Learning for Person Re-identification
• Convolution with even-sized kernels and symmetric padding
• Dynamic Neural Networks: A Survey
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• AxTrain: Hardware-Oriented Neural Network Training for Approximate Inference
• Training Binary Neural Networks through Learning with Noisy Supervision
• In-network Neural Networks
• Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence
• Controlling Information Capacity of Binary Neural Network
• Training Competitive Binary Neural Networks from Scratch
• Accurate and Compact Convolutional Neural Networks with Trained Binarization
• Efficient Exact Verification of Binarized Neural Networks
• QUOTIENT: Two-Party Secure Neural Network Training and Prediction
• Energy-efficient neural network inference with microcavity exciton-polaritons
• Streamlined Deployment for Quantized Neural Networks
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• Learning through atypical "phase transitions" in overparameterized neural networks
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• Kernel Based Progressive Distillation for Adder Neural Networks
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• Learning to Prune Filters in Convolutional Neural Networks
• Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
• Neural Network-Optimized Channel Estimator and Training Signal Design for MIMO Systems with Few-Bit ADCs
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Towards Effective Low-bitwidth Convolutional Neural Networks
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Path Planning using Neural A* Search
• Variation-aware Binarized Memristive Networks
• DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer
• Towards Efficient Processing and Learning with Spikes: New Approaches for Multi-Spike Learning
• Q-CapsNets: A Specialized Framework for Quantizing Capsule Networks
• Ultra-compact Binary Neural Networks for Human Activity Recognition on RISC-V Processors
• Resource-Efficient Neural Networks for Embedded Systems
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• Regularizing Activation Distribution for Training Binarized Deep Networks
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• A Dimensionality Reduction Approach for Convolutional Neural Networks
• HAWQV3: Dyadic Neural Network Quantization
• DeepHammer: Depleting the Intelligence of Deep Neural Networks through Targeted Chain of Bit Flips
• Deep Learning as a Mixed Convex-Combinatorial Optimization Problem
• Learned Low Precision Graph Neural Networks
• Incremental Binarization On Recurrent Neural Networks For Single-Channel Source Separation
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• 26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone
• Switchable Precision Neural Networks
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• MoBiNet: A Mobile Binary Network for Image Classification
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Continual 3D Convolutional Neural Networks for Real-time Processing of Videos
• Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive ADMM
• Generative Low-bitwidth Data Free Quantization
• CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long Way
• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• Learning Mesh Representations via Binary Space Partitioning Tree Networks
• Optimisation of the PointPillars network for 3D object detection in point clouds
• Deep learning via message passing algorithms based on belief propagation
• BSP-Net: Generating Compact Meshes via Binary Space Partitioning
• BoolNet: Minimizing The Energy Consumption of Binary Neural Networks
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition
• LIT: Block-wise Intermediate Representation Training for Model Compression
• Secure Quantized Training for Deep Learning
• Rethinking Differentiable Search for Mixed-Precision Neural Networks
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• BiPointNet: Binary Neural Network for Point Clouds
• Weight Normalization based Quantization for Deep Neural Network Compression
• Learning Deep Representations with Probabilistic Knowledge Transfer
• Mirror Descent View for Neural Network Quantization
• BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet
• VSA: Reconfigurable Vectorwise Spiking Neural Network Accelerator
• Learning Frequency Domain Approximation for Binary Neural Networks
• Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over Simplex
• Growing Efficient Deep Networks by Structured Continuous Sparsification
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
• Deep Molecular Programming: A Natural Implementation of Binary-Weight ReLU Neural Networks
• Towards Efficient Unconstrained Palmprint Recognition via Deep Distillation Hashing
• Network Quantization with Element-wise Gradient Scaling
• Boosting Binary Masks for Multi-Domain Learning through Affine Transformations
• Generalized Negative Correlation Learning for Deep Ensembling
• Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• ASTERYX : A model-Agnostic SaT-basEd appRoach for sYmbolic and score-based eXplanations
• Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networks
• ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks
• Multi-bin Trainable Linear Unit for Fast Image Restoration Networks
• BrainSlug: Transparent Acceleration of Deep Learning Through Depth-First Parallelism
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks
• On Tractable Representations of Binary Neural Networks
• Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
• Pruning Deep Neural Networks using Partial Least Squares
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Hazard Detection in Supermarkets using Deep Learning on the Edge
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking
• Weight Pruning via Adaptive Sparsity Loss
• Neural Logic Rule Layers
• Exploring the Back Alleys: Analysing The Robustness of Alternative Neural Network Architectures against Adversarial Attacks
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• Pufferfish: Communication-efficient Models At No Extra Cost
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
• Learning low-precision neural networks without Straight-Through Estimator(STE)
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• Binarizing MobileNet via Evolution-based Searching
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Direction is what you need: Improving Word Embedding Compression in Large Language Models
• From Shallow to Deep Interactions Between Knowledge Representation, Reasoning and Machine Learning (Kay R. Amel group)
• SOTERIA: In Search of Efficient Neural Networks for Private Inference
• Training Deep Neural Network in Limited Precision
• SeesawFaceNets: sparse and robust face verification model for mobile platform
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Pruning and Slicing Neural Networks using Formal Verification
• Lottery Jackpots Exist in Pre-trained Models
• Binary Complex Neural Network Acceleration on FPGA
• Efficient non-uniform quantizer for quantized neural network targeting reconfigurable hardware
• PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution
• Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training
• Binarized Convolutional Neural Networks with Separable Filters for Efficient Hardware Acceleration
• The Challenge of Multi-Operand Adders in CNNs on FPGAs: How not to solve it!
• A Review of Formal Methods applied to Machine Learning
• On Psychoacoustically Weighted Cost Functions Towards Resource-Efficient Deep Neural Networks for Speech Denoising
• Stein Variational Inference for Discrete Distributions
• PixelHop++: A Small Successive-Subspace-Learning-Based (SSL-based) Model for Image Classification
• Towards Explainable Bit Error Tolerance of Resistive RAM-Based Binarized Neural Networks
• Joint Multi-User DNN Partitioning and Computational Resource Allocation for Collaborative Edge Intelligence
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• Quantized Neural Networks for Radar Interference Mitigation
• Improving Binary Neural Networks through Fully Utilizing Latent Weights
• Compact representations of convolutional neural networks via weight pruning and quantization
• Dynamic Network Quantization for Efficient Video Inference
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• Neural Machine Translation with 4-Bit Precision and Beyond
• Learning Recurrent Binary/Ternary Weights
• DASNet: Dynamic Activation Sparsity for Neural Network Efficiency Improvement
• DNN Feature Map Compression using Learned Representation over GF(2)
• Local Binary Pattern Networks
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Balanced Binary Neural Networks with Gated Residual
• Dense xUnit Networks
• Fast Adjustable Threshold For Uniform Neural Network Quantization (Winning solution of LPIRC-II)
• HBONet: Harmonious Bottleneck on Two Orthogonal Dimensions
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning
• Universal Adder Neural Networks
• RecNets: Channel-wise Recurrent Convolutional Neural Networks
• Weight Equalizing Shift Scaler-Coupled Post-training Quantization
• Neural Networks Weights Quantization: Target None-retraining Ternary (TNT)
• AQD: Towards Accurate Fully-Quantized Object Detection
• NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing
• Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
• SiMaN: Sign-to-Magnitude Network Binarization
• Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent
• Always-On 674uW @ 4GOP/s Error Resilient Binary Neural Networks with Aggressive SRAM Voltage Scaling on a 22nm IoT End-Node
• Learning to Embed Categorical Features without Embedding Tables for Recommendation
• Magnetoresistive RAM for error resilient XNOR-Nets
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• BCNN: Binary Complex Neural Network
• Supported-BinaryNet: Bitcell Array-based Weight Supports for Dynamic Accuracy-Latency Trade-offs in SRAM-based Binarized Neural Network
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
• BABO: Background Activation Black-Out for Efficient Object Detection
• DP-Net: Dynamic Programming Guided Deep Neural Network Compression
• Scalable Approximate Inference and Some Applications
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory
• Making Logic Learnable With Neural Networks
• Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
• Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks
• BasisNet: Two-stage Model Synthesis for Efficient Inference
• Pruning Ternary Quantization
• Federated Learning via Plurality Vote
• Temporally Resolution Decrement: Utilizing the Shape Consistency for Higher Computational Efficiency
• A VM/Containerized Approach for Scaling TinyML Applications
• FATNN: Fast and Accurate Ternary Neural Networks
• Verifying Quantized Neural Networks using SMT-Based Model Checking
• Efficient and Robust Machine Learning for Real-World Systems
• Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks
• Finding Everything within Random Binary Networks
• Fully Quantized Image Super-Resolution Networks
• Digging Deeper into CRNN Model in Chinese Text Images Recognition
• Universal Approximation Theorems of Fully Connected Binarized Neural Networks
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Min-Max-Plus Neural Networks
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• BDD4BNN: A BDD-based Quantitative Analysis Framework for Binarized Neural Networks
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• Learning to map between ferns with differentiable binary embedding networks
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Low-memory convolutional neural networks through incremental depth-first processing
• Learning Effective Binary Visual Representations with Deep Networks
• TOCO: A Framework for Compressing Neural Network Models Based on Tolerance Analysis
• Quantized Neural Network Inference with Precision Batching
• On Transformations in Stochastic Gradient MCMC
• Toward Runtime-Throttleable Neural Networks
• Neural Parameter Allocation Search
• Hashing based Answer Selection
• GHFP: Gradually Hard Filter Pruning
• Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classification
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• End to End Binarized Neural Networks for Text Classification
• Compressing Deep Convolutional Neural Networks by Stacking Low-dimensional Binary Convolution Filters
• Block-term Tensor Neural Networks
• PINFER: Privacy-Preserving Inference for Machine Learning
• DNN Quantization with Attention
• MBS: Macroblock Scaling for CNN Model Reduction
• Smoothed Differential Privacy
• Quantized Proximal Averaging Network for Analysis Sparse Coding
• NodeDrop: A Condition for Reducing Network Size without Effect on Output
• Binarized Convolutional Neural Networks for Efficient Inference on GPUs
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
• Lossless Compression with Latent Variable Models
• Activation Map Adaptation for Effective Knowledge Distillation
• CodNN -- Robust Neural Networks From Coded Classification
• BRIEF: Backward Reduction of CNNs with Information Flow Analysis
• Kernel Quantization for Efficient Network Compression
• Scientific Calculator for Designing Trojan Detectors in Neural Networks
• PokeBNN: A Binary Pursuit of Lightweight Accuracy
• Carrying out CNN Channel Pruning in a White Box
• An Alternative Practice of Tropical Convolution to Traditional Convolutional Neural Networks
• Resource-Efficient Speech Mask Estimation for Multi-Channel Speech Enhancement
• Content-adaptive Representation Learning for Fast Image Super-resolution
• DTNN: Energy-efficient Inference with Dendrite Tree Inspired Neural Networks for Edge Vision Applications
• Robust error bounds for quantised and pruned neural networks
• LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes
• Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration
• TableNet: a multiplier-less implementation of neural networks for inferencing
• Multi-Glimpse Network: A Robust and Efficient Classification Architecture based on Recurrent Downsampled Attention
• Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations
• Distilling Knowledge From a Deep Pose Regressor Network
• Planning with Learned Binarized Neural Networks Benchmarks for MaxSAT Evaluation 2021
• Memory-Efficient Factorization Machines via Binarizing both Data and Model Coefficients
• GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond
• Random matrices in service of ML footprint: ternary random features with no performance loss
• Sparsity-Control Ternary Weight Networks
• CoCo DistillNet: a Cross-layer Correlation Distillation Network for Pathological Gastric Cancer Segmentation
• Binarized P-Network: Deep Reinforcement Learning of Robot Control from Raw Images on FPGA
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• Learned Image Compression for Machine Perception