Large Linguistic Models: Investigating LLMs' metalinguistic abilities

Abstract

The performance of large language models (LLMs) has recently improved to the point where models can perform well on many language tasks. We show here that--for the first time--the models can also generate valid metalinguistic analyses of language data. We outline a research program where the behavioral interpretability of LLMs on these tasks is tested via prompting. LLMs are trained primarily on text--as such, evaluating their metalinguistic abilities improves our understanding of their general capabilities and sheds new light on theoretical models in linguistics. We show that OpenAI's (2024) o1 vastly outperforms other models on tasks involving drawing syntactic trees and phonological generalization. We speculate that OpenAI o1's unique advantage over other models may result from the model's chain-of-thought mechanism, which mimics the structure of human reasoning used in complex cognitive tasks, such as linguistic analysis.

References in corpus (6)

• Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique
• The Effect of Sampling Temperature on Problem Solving in Large Language Models
• CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks
• Language models align with human judgments on key grammatical constructions
• Large Linguistic Models: Investigating LLMs' metalinguistic abilities
• Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data

Cited by in corpus (1)

• Large Linguistic Models: Investigating LLMs' metalinguistic abilities