I am a Strange Dataset: Metalinguistic Tests for Language Models

Tristan Thrush, Jared Moore, Miguel Monares +2

Statements involving metalinguistic self-reference ("This paper has six sections.") are prevalent in many domains. Can current large language models (LLMs) handle such language? In…