3 papers
cs.CL2023
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently…
cs.CL2023
Pseudointelligence: A Unifying Framework for Language Model Evaluation
Shikhar Murty, Orr Paradise, Pratyusha Sharma
With large language models surpassing human performance on an increasing number of benchmarks, we must take a principled approach for targeted evaluation of model capabilities. Ins…
cs.CL2023
Grokking of Hierarchical Structure in Vanilla Transformers
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
For humans, language production and comprehension is sensitive to the hierarchical structure of sentences. In natural language processing, past work has questioned how effectively…