4 citations · 8 across the 20 of their papers we have counts for
23 papers
Defining and Generating Compositional Structure
Eric Elmoznino, Tejas Kasetty, Blaise Agüera y Arcas +2
Compositional structure is pervasive in nature and across scales: atoms combine into molecules, molecules into cells, and cells into organisms, while further compositional hierarch…
KV-streams for Efficient Compaction in Agentic Reinforcement Learning
Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda +15
Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate…
Recursive Vision Language Models for General Symbolic Reasoning
Omid Nejati Manzari, Guillaume Lajoie, Hassan Rivaz
Hard symbolic-reasoning tasks such as Sudoku, maze pathfinding, and ARC remain challenging for LLMs due to their fixed-depth autoregressive reasoning, which limits systematic searc…
Can In-Context Learning Support Intrinsic Curiosity?
Eric Elmoznino, Sangnie Bhardwaj, Johannes von Oswald +5
Effective machine learning depends not only on how we model data, but also on what data we choose to collect. While large sequence models have revolutionized data modeling, the pro…
Simplifying the Modeling of Arbitrary Conditionals in Natural Language
Yinhan Lu, Eric Elmoznino, Léo Gagnon +3
Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood com…
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
Joachim Schaeffer, Thomas Jiralerspong, Alexander Panfilov +4
AI control protocols oversee untrusted models by monitoring their actions and modifying potentially unsafe steps, often using a trusted model. This partially tampers with the untru…
Causal Network Discovery from Interventional Count Data with Latent Linear DAGs
Yijiao Zhang, Hongzhe Li
The increasing availability of interventional data offers new opportunities for causal discovery, with gene perturbation studies providing a prominent example. Such data are typica…
Iterative Amortized Inference: Unifying In-Context Learning and Learned Optimizers
Sarthak Mittal, Divyat Mahajan, Guillaume Lajoie +1
Modern learning systems increasingly rely on amortized learning - the idea of reusing computation or inductive biases shared across tasks to enable rapid generalization to novel pr…
Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models
Siddarth Venkatraman, Vineet Jain, Sarthak Mittal +9
Test-time scaling methods improve the capabilities of large language models (LLMs) by increasing the amount of compute used during inference to make a prediction. Inference-time co…
Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective
Leo Gagnon, Eric Elmoznino, Sarthak Mittal +4
The rapid adaptation ability of auto-regressive foundation models is often attributed to the diversity of their pre-training data. This is because, from a Bayesian standpoint, mini…
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi +14
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compu…
In-Context Parametric Inference: Point or Distribution Estimators?
Sarthak Mittal, Yoshua Bengio, Nikolay Malkin +1
Bayesian and frequentist inference are two fundamental paradigms in statistical estimation. Bayesian methods treat hypotheses as random variables, incorporating priors and updating…
Amortized In-Context Bayesian Posterior Estimation
Sarthak Mittal, Niels Leif Bracher, Guillaume Lajoie +2
Bayesian inference provides a natural way of incorporating prior beliefs and assigning a probability measure to the space of hypotheses. Current solutions rely on iterative routine…
Multi-agent cooperation through learning-aware policy gradients
Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald +6
Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. How can we achieve cooperation among self-interested, independent lear…
A Complexity-Based Theory of Compositionality
Eric Elmoznino, Thomas Jiralerspong, Yoshua Bengio +1
Compositionality is believed to be fundamental to intelligence. In humans, it underlies the structure of thought, language, and higher-level reasoning. In AI, compositional represe…
In-context learning and Occam's razor
Eric Elmoznino, Tom Marty, Tejas Kasetty +5
A central goal of machine learning is generalization. While the No Free Lunch Theorem states that we cannot obtain theoretical guarantees for generalization without further assumpt…
Latent Representation Learning for Multimodal Brain Activity Translation
Arman Afrasiyabi, Dhananjay Bhaskar, Erica L. Busch +5
Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high tempora…
When can transformers compositionally generalize in-context?
Seijin Kobayashi, Simon Schug, Yassir Akram +5
Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during trainin…
Does learning the right latent variables necessarily improve in-context learning?
Sarthak Mittal, Eric Elmoznino, Leo Gagnon +4
Large autoregressive models like Transformers can solve tasks through in-context learning (ICL) without learning new weights, suggesting avenues for efficiently solving new tasks.…
Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
Tianhong Li, Sangnie Bhardwaj, Yonglong Tian +6
Current vision-language generative models rely on expansive corpora of paired image-text data to attain optimal performance and generalization capabilities. However, automatically…
Amortizing intractable inference in large language models
Edward J. Hu, Moksh Jain, Eric Elmoznino +4
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowled…
Delta-AI: Local objectives for amortized inference in sparse graphical models
Jean-Pierre Falet, Hae Beom Lee, Esmeralda S. Whitammer +6
We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call -amortized inference (-AI). Our approach is based on the obs…
Discrete, compositional, and symbolic representations through attractor dynamics
Andrew Nam, Eric Elmoznino, Nikolay Malkin +3
Symbolic systems are powerful frameworks for modeling cognitive processes as they encapsulate the rules and relationships fundamental to many aspects of human reasoning and behavio…