Trained Ternary Quantization

Abstract

Deep neural networks are widely used in machine learning applications. However, the deployment of large neural networks models can be difficult to deploy on mobile devices with limited power budgets. To solve this problem, we propose Trained Ternary Quantization (TTQ), a method that can reduce the precision of weights in neural networks to ternary values. This method has very little accuracy degradation and can even improve the accuracy of some models (32, 44, 56-layer ResNet) on CIFAR-10 and AlexNet on ImageNet. And our AlexNet model is trained from scratch, which means it's as easy as to train normal full precision model. We highlight our trained quantization method that can learn both ternary values and ternary assignment. During inference, only ternary values (2-bit weights) and scaling factors are needed, therefore our models are nearly 16x smaller than full-precision models. Our ternary models can also be viewed as sparse binary weight networks, which can potentially be accelerated with custom circuit. Experiments on CIFAR-10 show that the ternary models obtained by trained quantization method outperform full-precision models of ResNet-32,44,56 by 0.04%, 0.16%, 0.36%, respectively. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.

Accepted for Poster Presentation on ICLR 2017

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Identity Mappings in Deep Residual Networks
• Neural Networks with Few Multiplications

Cited by in corpus (278)

• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Bayesian Compression for Deep Learning
• Training and Inference with Integers in Deep Neural Networks
• Learned Step Size Quantization
• MCUNet: Tiny Deep Learning on IoT Devices
• Model compression via distillation and quantization
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• A Survey on Methods and Theories of Quantized Neural Networks
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• WRPN: Wide Reduced-Precision Networks
• Tiny Machine Learning: Progress and Futures
• MLPerf Training Benchmark
• Bringing AI To Edge: From Deep Learning's Perspective
• TSM: Temporal Shift Module for Efficient Video Understanding
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions
• A Survey of FPGA-Based Neural Network Accelerator
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Accelerating CNN inference on FPGAs: A Survey
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Loss-aware Weight Quantization of Deep Networks
• Visual Wake Words Dataset
• Applications and Techniques for Fast Machine Learning in Science
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• Dynamic Convolution: Attention over Convolution Kernels
• ProxQuant: Quantized Neural Networks via Proximal Operators
• DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Training Binary Neural Networks with Real-to-Binary Convolutions
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Constructing Accurate and Efficient Deep Spiking Neural Networks with Double-threshold and Augmented Schemes
• Ternary Neural Networks with Fine-Grained Quantization
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
• QKD: Quantization-aware Knowledge Distillation
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Back to Simplicity: How to Train Accurate BNNs from Scratch?
• TernaryNet: Faster Deep Model Inference without GPUs for Medical 3D Segmentation using Sparse and Binary Convolutions
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• A Survey of FPGA Based Deep Learning Accelerators: Challenges and Opportunities
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• Quantization of Fully Convolutional Networks for Accurate Biomedical Image Segmentation
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Low-Memory Neural Network Training: A Technical Report
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Defensive Quantization: When Efficiency Meets Robustness
• Towards Efficient Training for Neural Network Quantization
• DSConv: Efficient Convolution Operator
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments
• IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• Forward and Backward Information Retention for Accurate Binary Neural Networks
• Training DNNs with Hybrid Block Floating Point
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• MARS: Multi-macro Architecture SRAM CIM-Based Accelerator with Co-designed Compressed Neural Networks
• Neural Network-Optimized Channel Estimator and Training Signal Design for MIMO Systems with Few-Bit ADCs
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• Self-Binarizing Networks
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
• Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• Towards Effective Low-bitwidth Convolutional Neural Networks
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• Model Pruning Based on Quantified Similarity of Feature Maps
• BioNetExplorer: Architecture-Space Exploration of Bio-Signal Processing Deep Neural Networks for Wearables
• QGAN: Quantized Generative Adversarial Networks
• Distilled Neural Networks for Efficient Learning to Rank
• DeepTwist: Learning Model Compression via Occasional Weight Distortion
• A Programmable Approach to Neural Network Compression
• Resource-Efficient Neural Networks for Embedded Systems
• Deep Learning for Real-Time Crime Forecasting and its Ternarization
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• ZeroQ: A Novel Zero Shot Quantization Framework
• Joint Neural Architecture Search and Quantization
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Incremental Binarization On Recurrent Neural Networks For Single-Channel Source Separation
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Learned Low Precision Graph Neural Networks
• Deep Learning as a Mixed Convex-Combinatorial Optimization Problem
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Relaxed Quantization for Discretized Neural Networks
• FQ-Conv: Fully Quantized Convolution for Efficient and Accurate Inference
• Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
• Adaptive Precision CNN Accelerator Using Radix-X Parallel Connected Memristor Crossbars
• Vau da muntanialas: Energy-efficient multi-die scalable acceleration of RNN inference
• Knowledge Distillation for End-to-End Person Search
• MoBiNet: A Mobile Binary Network for Image Classification
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors
• Efficient Bitwidth Search for Practical Mixed Precision Neural Network
• Ternary Compression for Communication-Efficient Federated Learning
• Improving Branch Prediction By Modeling Global History with Convolutional Neural Networks
• Knowledge Squeezed Adversarial Network Compression
• DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics
• LIT: Block-wise Intermediate Representation Training for Model Compression
• Exploring Automatic Gym Workouts Recognition Locally On Wearable Resource-Constrained Devices
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• PruneNet: Channel Pruning via Global Importance
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• FleXOR: Trainable Fractional Quantization
• Learning on Hardware: A Tutorial on Neural Network Accelerators and Co-Processors
• Efficient Deep Neural Networks
• Precision Highway for Ultra Low-Precision Quantization
• Sparse Weight Activation Training
• Efficient Inference of CNNs via Channel Pruning
• Espresso: Efficient Forward Propagation for BCNNs
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
• Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework
• Some open questions on morphological operators and representations in the deep learning era
• SYMOG: learning symmetric mixture of Gaussian modes for improved fixed-point quantization
• Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• WRPN: Training and Inference using Wide Reduced-Precision Networks
• Learning Filter Basis for Convolutional Neural Network Compression
• MWQ: Multiscale Wavelet Quantized Neural Networks
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Regularized Binary Network Training
• On Compressing U-net Using Knowledge Distillation
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Towards Lossless Binary Convolutional Neural Networks Using Piecewise Approximation
• Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion
• Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
• Ternary Residual Networks
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• BitNet: Bit-Regularized Deep Neural Networks
• Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks
• QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware
• A 1Mb mixed-precision quantized encoder for image classification and patch-based compression
• Design Flow of Accelerating Hybrid Extremely Low Bit-width Neural Network in Embedded FPGA
• End-to-End Learned Image Compression with Quantized Weights and Activations
• Learning Architectures for Binary Networks
• DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization
• Pufferfish: Communication-efficient Models At No Extra Cost
• Binarizing MobileNet via Evolution-based Searching
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Toward Efficient Convolutional Neural Networks With Structured Ternary Patterns
• Build a Compact Binary Neural Network through Bit-level Sensitivity and Data Pruning
• RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• 3DQ: Compact Quantized Neural Networks for Volumetric Whole Brain Segmentation
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• MSP: An FPGA-Specific Mixed-Scheme, Multi-Precision Deep Neural Network Quantization Framework
• A Survey of Mobile Computing for the Visually Impaired
• Training Deep Neural Network in Limited Precision
• Quantization of Deep Neural Networks for Accurate Edge Computing
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• 8-bit Optimizers via Block-wise Quantization
• How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference
• Efficient non-uniform quantizer for quantized neural network targeting reconfigurable hardware
• An Investigation on Different Underlying Quantization Schemes for Pre-trained Language Models
• A flexible framework for communication-efficient machine learning: from HPC to IoT
• Post-Training Quantization for Vision Transformer
• Distributed Low Precision Training Without Mixed Precision
• FALCON: Lightweight and Accurate Convolution
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
• NASB: Neural Architecture Search for Binary Convolutional Neural Networks
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• When to Prune? A Policy towards Early Structural Pruning
• DiabDeep: Pervasive Diabetes Diagnosis based on Wearable Medical Sensors and Efficient Neural Networks
• Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
• High-Capacity Expert Binary Networks
• Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization
• DNQ: Dynamic Network Quantization
• Learning Recurrent Binary/Ternary Weights
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• HALP: Hardware-Aware Latency Pruning
• Network Automatic Pruning: Start NAP and Take a Nap
• Histogram-Equalized Quantization for logic-gated Residual Neural Networks
• Deep Neural Network Approximation using Tensor Sketching
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• Efficient Inferencing of Compressed Deep Neural Networks
• GDRQ: Group-based Distribution Reshaping for Quantization
• RTN: Reparameterized Ternary Network
• Tuning Algorithms and Generators for Efficient Edge Inference
• Learning Multimodal Fixed-Point Weights using Gradient Descent
• A Very Compact Embedded CNN Processor Design Based on Logarithmic Computing
• Structured Compression by Weight Encryption for Unstructured Pruning and Quantization
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• Dense xUnit Networks
• SGQuant: Squeezing the Last Bit on Graph Neural Networks with Specialized Quantization
• Scalar Arithmetic Multiple Data: Customizable Precision for Deep Neural Networks
• Integer-Only Neural Network Quantization Scheme Based on Shift-Batch-Normalization
• Going Beyond Classification Accuracy Metrics in Model Compression
• Quantized Neural Network Inference with Precision Batching
• Faster Secure Data Mining via Distributed Homomorphic Encryption
• FATNN: Fast and Accurate Ternary Neural Networks
• Analytical aspects of non-differentiable neural networks
• Efficient and Robust Machine Learning for Real-World Systems
• EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM
• Towards thinner convolutional neural networks through Gradually Global Pruning
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• Finding Everything within Random Binary Networks
• Pruning Ternary Quantization
• Improving Network Slimming with Nonconvex Regularization
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization
• Low-memory convolutional neural networks through incremental depth-first processing
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• Information-Theoretic Understanding of Population Risk Improvement with Model Compression
• $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing
• Sparsity-Control Ternary Weight Networks
• Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classification
• Robustness-aware 2-bit quantization with real-time performance for neural network
• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond
• AutoQNN: An End-to-End Framework for Automatically Quantizing Neural Networks
• Improving DNN Fault Tolerance using Weight Pruning and Differential Crossbar Mapping for ReRAM-based Edge AI
• Variational Dropout Sparsification for Particle Identification speed-up
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
• Adaptive Binary-Ternary Quantization
• CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• Adaptive Precision Training for Resource Constrained Devices
• Hybrid and Non-Uniform quantization methods using retro synthesis data for efficient inference
• Light Multi-segment Activation for Model Compression
• ViP: Virtual Pooling for Accelerating CNN-based Image Classification and Object Detection
• DNN Quantization with Attention
• Faster Convolution Inference Through Using Pre-Calculated Lookup Tables
• Cross-filter compression for CNN inference acceleration
• Self-grouping Convolutional Neural Networks
• Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• Cluster Regularized Quantization for Deep Networks Compression
• $S^2$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning
• Full-stack Optimization for Accelerating CNNs with FPGA Validation
• A Targeted Acceleration and Compression Framework for Low bit Neural Networks
• Binarized Weight Error Networks With a Transition Regularization Term