Training Transformers with 4-bit Integers

Haocheng Xi, Changhao Li, Jianfei Chen +1

Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical forma…