On Periodic Functions as Regularizers for Quantization of Neural Networks

Abstract

Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply these functions component-wise and add the sum over the model parameters as a regularizer to the model loss during training. The frequency and amplitude hyper-parameters of these functions can be adjusted during training. The regularization pushes the weights into discrete points that can be encoded as integers. We show that using this technique the resulting quantized models exhibit the same accuracy as the original ones on CIFAR-10 and ImageNet datasets.

arXiv comment11 pages, 7 figures

References in corpus (14)

• Deep Residual Learning for Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Going Deeper with Convolutions
• Compressing Deep Convolutional Networks using Vector Quantization
• Ternary Weight Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Adaptive Quantization for Deep Neural Network
• Training Bit Fully Convolutional Network for Fast Semantic Segmentation

Cited by in corpus (7)

• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Dustin: A 16-Cores Parallel Ultra-Low-Power Cluster with 2b-to-32b Fully Flexible Bit-Precision and Vector Lockstep Execution Mode
• HAWQV3: Dyadic Neural Network Quantization
• Mixed-Precision Embedding Using a Cache
• Training with Multi-Layer Embeddings for Model Reduction
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization