Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks

Abstract

The use of deep neural networks in edge computing devices hinges on the balance between accuracy and complexity of computations. Ternary Connect (TC) \cite{lin2015neural} addresses this issue by restricting the parameters to three levels $-1,0$, and $+1$, thus eliminating multiplications in the forward pass of the network during prediction. We propose Generalized Ternary Connect (GTC), which allows an arbitrary number of levels while at the same time eliminating multiplications by restricting the parameters to integer powers of two. The primary contribution is that GTC learns the number of levels and their values for each layer, jointly with the weights of the network in an end-to-end fashion. Experiments on MNIST and CIFAR-10 show that GTC naturally converges to an `almost binary' network for deep classification networks (e.g. VGG-16) and deep variational auto-encoders, with negligible loss of classification accuracy and comparable visual quality of generated samples respectively. We demonstrate superior compression and similar accuracy of GTC in comparison to several state-of-the-art methods for neural network compression. We conclude with simulations showing the potential benefits of GTC in hardware.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Convolutional Neural Networks using Logarithmic Data Representation
• Bayesian Compression for Deep Learning
• Training and Inference with Integers in Deep Neural Networks
• Model compression via distillation and quantization
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Neural Networks with Few Multiplications
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• BitNet: Bit-Regularized Deep Neural Networks
• WRPN & Apprentice: Methods for Training and Inference using Low-Precision Numerics

Cited by in corpus (2)

• Bit Efficient Quantization for Deep Neural Networks
• Toward Runtime-Throttleable Neural Networks