Exploring Quantization for Efficient Pre-Training of Transformer Language Models

Kamran Chitsaz, Quentin Fournier, Gonçalo Mordido +1

The increasing scale of Transformer models has led to an increase in their pre-training computational requirements. While quantization has proven to be effective after pre-training…