How Large Language Models Get Stuck: Early structure with persistent errors

Alokesh Manna, William Snyder, Whitney Tabor

Linguistic insights may help make Large Language Model (LLM) training more efficient. We trained Meta's OPT model on the 100M word BabyLM dataset, and evaluated it on the BLiMP ben…