21 citations · 27 across the 6 of their papers we have counts for
10 papers
TRA: Better Length Generalisation with Threshold Relative Attention
Mattia Opper, Roland Fernandez, Paul Smolensky +1
Transformers struggle with length generalisation, displaying poor performance even on basic tasks. We test whether these limitations can be explained through two key failures of th…
Compositional Generalization Across Distributional Shifts with Sparse Tree Operations
Paul Soulos, Henry Conklin, Mattia Opper +3
Neural networks continue to struggle with compositional generalization, and this issue is exacerbated by a lack of massive pre-training. One successful approach for developing neur…
Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks
Paul Smolensky, Roland Fernandez, Zhenghao Herbert Zhou +3
Large Language Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of critiques a…
Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems
Paul Smolensky, R. Thomas McCoy, Roland Fernandez +2
What explains the dramatic progress from 20th-century to 21st-century AI, and how can the remaining limitations of current AI be overcome? The widely accepted narrative attributes…
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
Yichen Jiang, Asli Celikyilmaz, Paul Smolensky +7
Abstractive summarization, the task of generating a concise summary of input documents, requires: (1) reasoning over the source document to determine the salient pieces of informat…
Compositional Processing Emerges in Neural Networks Solving Math Problems
Jacob Russin, Roland Fernandez, Hamid Palangi +4
A longstanding question in cognitive science concerns the learning mechanisms underlying compositionality in human cognition. Humans can infer the structured relationships (e.g., g…