Pre-training under infinite compute

Konwoo Kim, Suhas Kotha, Percy Liang +1

Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We…