How Syntax Specialization Emerges in Language Models

Xufeng Duan, Zhaoqian Yao, Yunhao Zhang +2

Large language models (LLMs) have been found to develop surprising internal specializations: Individual neurons, attention heads, and circuits become selectively sensitive to synta…