Zero-Shot Dynamic Quantization for Transformer Inference

Yousef El-Kurdi, Jerry Quinn, Avirup Sil

We introduce a novel run-time method for significantly reducing the accuracy loss associated with quantizing BERT-like models to 8-bit integers. Existing methods for quantizing mod…