Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalization

Tian Qin, Naomi Saphra, David Alvarez-Melis

Early in training, LMs can behave like n-gram models, but eventually they often learn tree-based syntactic rules and generalize hierarchically out of distribution (OOD). We study t…