Towards Efficient Post-training Quantization of Pre-trained Language Models

Abstract

Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow quantization-aware training~(QAT) that requires end-to-end training with full access to the entire dataset. Therefore, they suffer from slow training, large memory overhead, and data security issues. In this paper, we study post-training quantization~(PTQ) of PLMs, and propose module-wise quantization error minimization~(MREM), an efficient solution to mitigate these issues. By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module. In addition, we design a new model parallel training strategy such that each module can be trained locally on separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., $4\times$ on $4$ GPUs). Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• DARTS: Differentiable Architecture Search
• Zero-Shot Text-to-Image Generation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Ternary Weight Networks
• The Evolved Transformer
• Discrimination-aware Network Pruning for Deep Model Compression
• Training with Quantization Noise for Extreme Model Compression
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• Structured Pruning of a BERT-based Question Answering Model
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Distilling Knowledge by Mimicking Features
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism
• Go Wide, Then Narrow: Efficient Training of Deep Thin Networks

Cited by in corpus (6)

• Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation
• Towards Accurate Post-Training Quantization for Vision Transformer
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data