1 paper
Alexander Stern, Tim Rocktäschel, Acyr Locatelli +1
We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT…