Post-Training 4-bit Quantization on Embedding Tables

Abstract

Continuous representations have been widely adopted in recommender systems where a large number of entities are represented using embedding vectors. As the cardinality of the entities increases, the embedding components can easily contain millions of parameters and become the bottleneck in both storage and inference due to large memory consumption. This work focuses on post-training 4-bit quantization on the continuous embeddings. We propose row-wise uniform quantization with greedy search and codebook-based quantization that consistently outperforms state-of-the-art quantization approaches on reducing accuracy degradation. We deploy our uniform quantization technique on a production model in Facebook and demonstrate that it can reduce the model size to only 13.89% of the single-precision version while the model quality stays neutral.

Accepted in MLSys@NeurIPS'19 (http://learningsys.org/neurips19/)

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Mixed Precision Training
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Resiliency of Deep Neural Networks under Quantization
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• High-Accuracy Low-Precision Training
• Effective Quantization Methods for Recurrent Neural Networks

Cited by in corpus (9)

• Embedding Compression in Recommender Systems: A Survey
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• First-Generation Inference Accelerator Deployment at Facebook
• Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models
• Mixed-Precision Embedding Using a Cache
• Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• Supporting Massive DLRM Inference Through Software Defined Memory
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression