47 citations · 262 across the 33 of their papers we have counts for
33 papers
The Emergent Symbolic Structure of Artificial Neural Networks
R. Thomas McCoy, Paul Soulos, Tal Linzen +1
Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured…
GFlowRL: Scaling Distribution-Matching RL to Large Language Models
Xiaodong Liu, Michael Xu, Jack W. Stokes +3
Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths…
TRA: Better Length Generalisation with Threshold Relative Attention
Mattia Opper, Roland Fernandez, Paul Smolensky +1
Transformers struggle with length generalisation, displaying poor performance even on basic tasks. We test whether these limitations can be explained through two key failures of th…
Compositional Generalization Across Distributional Shifts with Sparse Tree Operations
Paul Soulos, Henry Conklin, Mattia Opper +3
Neural networks continue to struggle with compositional generalization, and this issue is exacerbated by a lack of massive pre-training. One successful approach for developing neur…
Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks
Paul Smolensky, Roland Fernandez, Zhenghao Herbert Zhou +3
Large Language Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of critiques a…
Implicit Chain of Thought Reasoning via Knowledge Distillation
Yuntian Deng, Kiran Prasad, Roland Fernandez +3
To augment language models with the ability to reason, researchers usually prompt or finetune them to produce chain of thought reasoning steps before producing the final answer. Ho…
Differentiable Tree Operations Promote Compositional Generalization
Paul Soulos, Edward Hu, Kate McCurdy +4
In the context of structure-to-structure transformation tasks, learning sequences of discrete symbolic operations poses significant challenges due to their non-differentiability. T…
Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models
Najoung Kim, Tal Linzen, Paul Smolensky
Human linguistic capacity is often characterized by compositionality and the generalization it enables -- human learners can produce and comprehend novel complex expressions by com…
Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
Paul Soulos, Sudha Rao, Caitlin Smith +9
Machine translation has seen rapid progress with the advent of Transformer-based models. These models have no explicit linguistic structure built into them, yet they may still impl…
Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems
Paul Smolensky, R. Thomas McCoy, Roland Fernandez +2
What explains the dramatic progress from 20th-century to 21st-century AI, and how can the remaining limitations of current AI be overcome? The widely accepted narrative attributes…
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN
R. Thomas McCoy, Paul Smolensky, Tal Linzen +2
Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apa…
Distributed neural encoding of binding to thematic roles
Matthias Lalisse, Paul Smolensky
A framework and method are proposed for the study of constituent composition in fMRI. The method produces estimates of neural patterns encoding complex linguistic structures, under…
Scalable knowledge base completion with superposition memories
Matthias Lalisse, Eric Rosen, Paul Smolensky
We present Harmonic Memory Networks (HMem), a neural architecture for knowledge base completion that models entities as weighted sums of pairwise bindings between an entity's neigh…
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
Yichen Jiang, Asli Celikyilmaz, Paul Smolensky +7
Abstractive summarization, the task of generating a concise summary of input documents, requires: (1) reasoning over the source document to determine the salient pieces of informat…
Compositional Processing Emerges in Neural Networks Solving Math Problems
Jacob Russin, Roland Fernandez, Hamid Palangi +4
A longstanding question in cognitive science concerns the learning mechanisms underlying compositionality in human cognition. Humans can infer the structured relationships (e.g., g…
Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
Hassan Akbari, Hamid Palangi, Jianwei Yang +6
Neuro-symbolic representations have proved effective in learning structure information in vision and language. In this paper, we propose a new model architecture for learning multi…
Universal linguistic inductive biases via meta-learning
R. Thomas McCoy, Erin Grant, Paul Smolensky +2
How do learners acquire languages from the limited data available to them? This process must involve some inductive biases - factors that affect how a learner generalizes - but it…
HUBERT Untangles BERT to Improve Transfer across NLP Tasks
Mehrad Moradshahi, Hamid Palangi, Monica S. Lam +2
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model.…
Discovering the Compositional Structure of Vector Representations with Role Learning Networks
Paul Soulos, Tom McCoy, Tal Linzen +1
How can neural networks perform so well on compositional tasks even though they lack explicit compositional representations? We use a novel analysis technique called ROLE to show t…
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
Imanol Schlag, Paul Smolensky, Roland Fernandez +3
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (T…
Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
Kezhen Chen, Qiuyuan Huang, Hamid Palangi +3
Generating formal-language programs represented by relational tuples, such as Lisp programs or mathematical operations, to solve problems stated in natural language is a challengin…
Quantum Language Processing
Nathan Wiebe, Alex Bocharov, Paul Smolensky +2
We present a representation for linguistic structure that we call a Fock-space representation, which allows us to embed problems in language processing into small quantum devices.…
RNNs Implicitly Implement Tensor Product Representations
R. Thomas McCoy, Tal Linzen, Ewan Dunbar +1
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regu…
Augmenting Compositional Models for Knowledge Base Completion Using Gradient Representations
Matthias Lalisse, Paul Smolensky
Neural models of Knowledge Base data have typically employed compositional representations of graph objects: entity and relation embeddings are systematically combined to evaluate…