13 citations · 13 across the 4 of their papers we have counts for
4 papers
Grokking of Hierarchical Structure in Vanilla Transformers
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
For humans, language production and comprehension is sensitive to the hierarchical structure of sentences. In natural language processing, past work has questioned how effectively…
Characterizing Intrinsic Compositionality in Transformers with Tree Projections
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
When trained on language data, do transformers learn some arbitrary computation that utilizes the full capacity of the architecture or do they learn a simpler, tree-like computatio…
How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction
Anthony Bau, Jacob Andreas
After a neural sequence model encounters an unexpected token, can its behavior be predicted? We show that RNN and transformer language models exhibit structured, consistent general…
Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales
Jacob Andreas, Gašper Beguš, Michael M. Bronstein +16
The past decade has witnessed a groundbreaking rise of machine learning for human language analysis, with current methods capable of automatically accurately recovering various asp…