Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Abstract

We introduce a method to train Quantized Neural Networks (QNNs) --- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train-time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.

arXiv commentarXiv admin note: text overlap with arXiv:1602.02830

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Deeply-Supervised Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• One weird trick for parallelizing convolutional neural networks
• Deep Learning using Linear Support Vector Machines
• Learning both Weights and Connections for Efficient Neural Networks
• Training deep neural networks with low precision multiplications
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Hardware-oriented Approximation of Convolutional Neural Networks
• Bitwise Neural Networks
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Estimating or Propagating Gradients Through Stochastic Neurons
• Deep neural networks are robust to weight binarization and other non-linear distortions
• Recurrent Neural Networks With Limited Numerical Precision

Cited by in corpus (362)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Mixed Precision Training
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Deep Learning Methods for Improved Decoding of Linear Codes
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• A Microprocessor implemented in 65nm CMOS with Configurable and Bit-scalable Accelerator for Programmable In-memory Computing
• CardioXNet: A Novel Lightweight Deep Learning Framework for Cardiovascular Disease Classification Using Heart Sound Recordings
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Scalable Methods for 8-bit Training of Neural Networks
• A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions
• Network Pruning via Transformable Architecture Search
• PULP-NN: Accelerating Quantized Neural Networks on Parallel Ultra-Low-Power RISC-V Processors
• TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
• Neural Network Approximation: Three Hidden Layers Are Enough
• Deep Learning-Based Model Predictive Control for Resonant Power Converters
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Mixed-precision deep learning based on computational memory
• Exploring the Connection Between Binary and Spiking Neural Networks
• Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• Review of data analysis in vision inspection of power lines with an in-depth discussion of deep learning technology
• Scaling Up Silicon Photonic-based Accelerators: Challenges and Opportunities
• Energy-efficient stochastic computing with superparamagnetic tunnel junctions
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• What Do Compressed Deep Neural Networks Forget?
• Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms
• Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
• Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• Scaling Deep Learning on GPU and Knights Landing clusters
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Efficiency of quantum versus classical annealing in non-convex learning problems
• Design Space Exploration of Neural Network Activation Function Circuits
• Natural Compression for Distributed Deep Learning
• Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Ternary Neural Networks with Fine-Grained Quantization
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• pLUTo: Enabling Massively Parallel Computation in DRAM via Lookup Tables
• A neural network memory prefetcher using semantic locality
• TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Charged particle tracking via edge-classifying interaction networks
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks
• DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
• Mixed Precision DNNs: All you need is a good parametrization
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• Towards Neural Mixture Recommender for Long Range Dependent User Sequences
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Tensorized Embedding Layers for Efficient Model Compression
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• RAPIDNN: In-Memory Deep Neural Network Acceleration Framework
• Unsupervised Visual Feature Learning with Spike-timing-dependent Plasticity: How Far are we from Traditional Feature Learning Approaches?
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Extending the limit of molecular dynamics with ab initio accuracy to 10 billion atoms
• Mixed Precision Training With 8-bit Floating Point
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• Quantization of Fully Convolutional Networks for Accurate Biomedical Image Segmentation
• More is Less: A More Complicated Network with Less Inference Complexity
• Low-Memory Neural Network Training: A Technical Report
• Knowledge Distillation via Route Constrained Optimization
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• DARKSIDE: A Heterogeneous RISC-V Compute Cluster for Extreme-Edge On-Chip DNN Inference and Training
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• DSConv: Efficient Convolution Operator
• Controlling Information Capacity of Binary Neural Network
• Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Energy Efficient Learning with Low Resolution Stochastic Domain Wall Synapse Based Deep Neural Networks
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
• Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments
• Towards Transmission-Friendly and Robust CNN Models over Cloud and Device
• Feature Map Transform Coding for Energy-Efficient CNN Inference
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• How Does Batch Normalization Help Binary Training?
• DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing
• Boosting Neural Machine Translation
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Knowledge Distillation from Internal Representations
• Optimizing Bit-Serial Matrix Multiplication for Reconfigurable Computing
• DeepTwist: Learning Model Compression via Occasional Weight Distortion
• QGAN: Quantized Generative Adversarial Networks
• Distilled Neural Networks for Efficient Learning to Rank
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• Ultra-compact Binary Neural Networks for Human Activity Recognition on RISC-V Processors
• Toward Scalable Verification for Safety-Critical Deep Networks
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• DeepHammer: Depleting the Intelligence of Deep Neural Networks through Targeted Chain of Bit Flips
• BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies
• Binarized Neural Networks on the ImageNet Classification Task
• Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• Compressive Sensing Using Iterative Hard Thresholding with Low Precision Data Representation: Theory and Applications
• To compress or not to compress: Understanding the Interactions between Adversarial Attacks and Neural Network Compression
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• Quantization of Deep Neural Networks for Accumulator-constrained Processors
• FQ-Conv: Fully Quantized Convolution for Efficient and Accurate Inference
• Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
• Iteratively Training Look-Up Tables for Network Quantization
• Relaxed Quantization for Discretized Neural Networks
• Automatic Mixed-Precision Quantization Search of BERT
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Backprop with Approximate Activations for Memory-efficient Network Training
• A scalable saliency-based Feature selection method with instance level information
• Taxonomy and Benchmarking of Precision-Scalable MAC Arrays Under Enhanced DNN Dataflow Representation
• Low precision logarithmic number systems: Beyond base-2
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• Additive Noise Annealing and Approximation Properties of Quantized Neural Networks
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• Mirage: An RNS-Based Photonic Accelerator for DNN Training
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Rethinking Bottleneck Structure for Efficient Mobile Network Design
• EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks
• Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Benchmarking Quantized Neural Networks on FPGAs with FINN
• Efficient Bitwidth Search for Practical Mixed Precision Neural Network
• On Periodic Functions as Regularizers for Quantization of Neural Networks
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• LIT: Block-wise Intermediate Representation Training for Model Compression
• LCNN: Lookup-based Convolutional Neural Network
• Fix your classifier: the marginal value of training the last weight layer
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Low-bit Quantization for Deep Graph Neural Networks with Smoothness-aware Message Propagation
• Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks
• Resource-Frugal Classification and Analysis of Pathology Slides Using Image Entropy
• Minimal Learning Machine: Theoretical Results and Clustering-Based Reference Point Selection
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
• All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
• PruneNet: Channel Pruning via Global Importance
• Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Mirror Descent View for Neural Network Quantization
• On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Optimal training of integer-valued neural networks with mixed integer programming
• ApproxNet: Content and Contention-Aware Video Analytics System for Embedded Clients
• Full deep neural network training on a pruned weight budget
• Learning in the Machine: Random Backpropagation and the Deep Learning Channel
• Training Compact Neural Networks with Binary Weights and Low Precision Activations
• Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over Simplex
• A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks
• Faster Neural Network Training with Approximate Tensor Operations
• An Energy-efficient Time-domain Analog VLSI Neural Network Processor Based on a Pulse-width Modulation Approach
• Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance
• AI in Space for Scientific Missions: Strategies for Minimizing Neural-Network Model Upload
• Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization
• Lightweight Convolutional Representations for On-Device Natural Language Processing
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework
• Boosting Binary Masks for Multi-Domain Learning through Affine Transformations
• Improving Adversarial Robustness in Weight-quantized Neural Networks
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Revisiting BFloat16 Training
• Real-time Mask Detection on Google Edge TPU
• Automating Generation of Low Precision Deep Learning Operators
• Multi-bin Trainable Linear Unit for Fast Image Restoration Networks
• BARVINN: Arbitrary Precision DNN Accelerator Controlled by a RISC-V CPU
• ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles
• TaxoNN: A Light-Weight Accelerator for Deep Neural Network Training
• Compressibility Loss for Neural Network Weights
• Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• Collaborative Distillation for Ultra-Resolution Universal Style Transfer
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Efficient Hybrid Network Architectures for Extremely Quantized Neural Networks Enabling Intelligence at the Edge
• Regularized Binary Network Training
• Weight Pruning via Adaptive Sparsity Loss
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
• Topology of deep neural networks
• Demystifying the MLPerf Benchmark Suite
• Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations
• Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
• Neural Epitome Search for Architecture-Agnostic Network Compression
• Pufferfish: Communication-efficient Models At No Extra Cost
• Hessian-Aware Pruning and Optimal Neural Implant
• BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
• Scalar Quantization as Sparse Least Square Optimization
• Generative Zero-shot Network Quantization
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Learning compositional functions via multiplicative weight updates
• BitNet: Bit-Regularized Deep Neural Networks
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Mixed-Precision Quantized Neural Network with Progressively Decreasing Bitwidth For Image Classification and Object Detection
• LP-3DCNN: Unveiling Local Phase in 3D Convolutional Neural Networks
• On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks
• Quantized Non-Volatile Nanomagnetic Synapse based Autoencoder for Efficient Unsupervised Network Anomaly Detection
• Synaptic metaplasticity with multi-level memristive devices
• Dynamic Runtime Feature Map Pruning
• DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization
• Training Energy-Efficient Deep Spiking Neural Networks with Time-to-First-Spike Coding
• Statistical Model Compression for Small-Footprint Natural Language Understanding
• SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
• CascadeCNN: Pushing the performance limits of quantisation
• Quantization of Deep Neural Networks for Accurate Edge Computing
• Low-complexity Recurrent Neural Network-based Polar Decoder with Weight Quantization Mechanism
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• Approximations in Deep Learning
• Efficient non-uniform quantizer for quantized neural network targeting reconfigurable hardware
• Knowledge distillation for optimization of quantized deep neural networks
• Inference with Hybrid Bio-hardware Neural Networks
• Convolutional Neural Network Quantization using Generalized Gamma Distribution
• Training Deep Neural Network in Limited Precision
• Pruning and Slicing Neural Networks using Formal Verification
• No Multiplication? No Floating Point? No Problem! Training Networks for Efficient Inference
• GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
• Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networks
• PrivPy: Enabling Scalable and General Privacy-Preserving Machine Learning
• Compressing Language Models using Doped Kronecker Products
• How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection
• Big-Little Adaptive Neural Networks on Low-Power Near-Subthreshold Processors
• Learning Recurrent Binary/Ternary Weights
• Ultra-Lightweight Speech Separation via Group Communication
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• Correlation Congruence for Knowledge Distillation
• Fast Video Classification via Adaptive Cascading of Deep Models
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• Image Captioning with Sparse Recurrent Neural Network
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• Distributed Low Precision Training Without Mixed Precision
• Histogram-Equalized Quantization for logic-gated Residual Neural Networks
• Neural Machine Translation with 4-Bit Precision and Beyond
• Impostor Networks for Fast Fine-Grained Recognition
• Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation
• Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach
• Compact representations of convolutional neural networks via weight pruning and quantization
• Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations
• UnitedQA: A Hybrid Approach for Open Domain Question Answering
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
• Progressive Learning of Low-Precision Networks
• Training of Quantized Deep Neural Networks using a Magnetic Tunnel Junction-Based Synapse
• Recurrent Convolution for Compact and Cost-Adjustable Neural Networks: An Empirical Study
• Deep Learning in Memristive Nanowire Networks
• The Pitfall of Evaluating Performance on Emerging AI Accelerators
• Spintronics for neuromorphic computing
• ESPN: Extremely Sparse Pruned Networks
• Distributed Learning on Heterogeneous Resource-Constrained Devices
• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization
• Representation Edit Distance as a Measure of Novelty
• A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness
• Integer-Only Neural Network Quantization Scheme Based on Shift-Batch-Normalization
• BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference
• The Tensor Track VII: From Quantum Gravity to Artificial Intelligence
• Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
• Active multi-fidelity Bayesian online changepoint detection
• Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
• SMYRF: Efficient Attention using Asymmetric Clustering
• Training Sparse Neural Networks using Compressed Sensing
• Channel Pruning via Optimal Thresholding
• Structured Compression by Weight Encryption for Unstructured Pruning and Quantization
• Dense xUnit Networks
• Recent Advances in Convolutional Neural Network Acceleration
• BISMO: A Scalable Bit-Serial Matrix Multiplication Overlay for Reconfigurable Computing
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• On the Universal Approximation Property and Equivalence of Stochastic Computing-based Neural Networks and Binary Neural Networks
• An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis
• Efficient Computation of Positional Population Counts Using SIMD Instructions
• LANCE: Efficient Low-Precision Quantized Winograd Convolution for Neural Networks Based on Graphics Processing Units
• Training of mixed-signal optical convolutional neural network with reduced quantization level
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization
• Quantized Artificial Neural Networks Implemented with Spintronic Stochastic Computing
• Learning In Practice: Reasoning About Quantization
• MOGNET: A Mux-residual quantized Network leveraging Online-Generated weights
• Verifying Quantized Neural Networks using SMT-Based Model Checking
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
• Efficient Inference via Universal LSH Kernel
• A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration
• Low-memory convolutional neural networks through incremental depth-first processing
• Finding Everything within Random Binary Networks
• Scalable Smartphone Cluster for Deep Learning
• Analytical aspects of non-differentiable neural networks
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• Training DNNs in O(1) memory with MEM-DFA using Random Matrices
• Universal Approximation Theorems of Fully Connected Binarized Neural Networks
• Enhancing efficiency of object recognition in different categorization levels by reinforcement learning in modular spiking neural networks
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification
• EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM
• Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation
• TOCO: A Framework for Compressing Neural Network Models Based on Tolerance Analysis
• Quantized Neural Network Inference with Precision Batching
• An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation
• Efficient and Robust Machine Learning for Real-World Systems
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• SpotPatch: Parameter-Efficient Transfer Learning for Mobile Object Detection
• Context-Free TextSpotter for Real-Time and Mobile End-to-End Text Detection and Recognition
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration
• Light Multi-segment Activation for Model Compression
• NN2CAM: Automated Neural Network Mapping for Multi-Precision Edge Processing on FPGA-Based Cameras
• Stochastic Computing for Hardware Implementation of Binarized Neural Networks
• Differentiable Architecture Pruning for Transfer Learning
• Associative Convolutional Layers
• Identifying and Exploiting Structures for Reliable Deep Learning
• Hardware realisation of nonlinear dynamical systems for and from biology
• 4-bit Quantization of LSTM-based Speech Recognition Models
• VC dimension of partially quantized neural networks in the overparametrized regime
• Magnitude and Uncertainty Pruning Criterion for Neural Networks
• The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
• Knowledge Distillation By Sparse Representation Matching
• Enabling Incremental Training with Forward Pass for Edge Devices
• Demystifying and Generalizing BinaryConnect
• Low Power In-Memory Implementation of Ternary Neural Networks with Resistive RAM-Based Synapse
• CloudifierNet -- Deep Vision Models for Artificial Image Processing
• Compression of Acoustic Event Detection Models With Quantized Distillation
• Maximin Optimization for Binary Regression
• Training for Speech Recognition on Coprocessors
• Partial Weight Adaptation for Robust DNN Inference
• Counting Cards: Exploiting Variance and Data Distributions for Robust Compute In-Memory
• XpulpNN: Enabling Energy Efficient and Flexible Inference of Quantized Neural Network on RISC-V based IoT End Nodes
• A Greedy Algorithm for Quantizing Neural Networks
• A Mixed-Precision RISC-V Processor for Extreme-Edge DNN Inference
• Voltage Controlled Energy Efficient Domain Wall Synapses with Stochastic Distribution of Quantized Weights in the Presence of Thermal Noise and Edge Roughness
• Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classification
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing
• Weakly Supervised Recovery of Semantic Attributes
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• Graphs for deep learning representations
• Quantized Proximal Averaging Network for Analysis Sparse Coding
• Binarized Weight Error Networks With a Transition Regularization Term
• Smoothed Differential Privacy
• KML: Using Machine Learning to Improve Storage Systems
• A Survey on Green Deep Learning
• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers
• Modulated binary cliquenet
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Improving deep neural network performance through sampling
• Adaptive Precision Training for Resource Constrained Devices
• A Tapered Floating Point Extension for the Redundant Signed Radix 2 System Using the Canonical Recoding
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks