A circuit for predicting hierarchical structure in-context in Large Language Models

Tankred Saanum, Can Demircan, Samuel J. Gershman +1

Large Language Models (LLMs) excel at in-context learning, the ability to use information provided as context to improve prediction of future tokens. Induction heads have been argu…