Benchmarking down-scaled (not so large) pre-trained language models

M. Aßenmacher, P. Schulze, C. Heumann

Large Transformer-based language models are pre-trained on corpora of varying sizes, for a different number of steps and with different batch sizes. At the same time, more fundamen…