Controlled Randomness Improves the Performance of Transformer Models

Tobias Deußer, Cong Zhao, Wolfgang Krämer +3

During the pre-training step of natural language models, the main objective is to learn a general representation of the pre-training dataset, usually requiring large amounts of tex…