Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1

Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently…