16 citations · 27 across the 25 of their papers we have counts for
26 papers
Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulence
Payel Mukhopadhyay, Stefan S. Nixon, Romain Watteaux +20
Whether physics foundation models can be usefully deployed on laboratory experiments remains an open question for scientific machine learning (ML). We test this question on the Ray…
Assign and Add: A Mechanistic Study of Compositional Arithmetic
Brady Exoo, Alberto Bietti, John Sous
Large language models are able to compose skills in order to perform complex tasks, many of which might not have been seen during training. The details of how exactly this composit…
Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation
Roman Klypa, Alberto Bietti, Sergei Grudinin
The design of RNA molecules that interact with specific proteins is a critical challenge in experimental and computational biology. Despite recent progress in natural language mode…
Geometric Factual Recall in Transformers
Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1
How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter…
MIMIC: A Generative Multimodal Foundation Model for Biomolecules
Siavash Golkar, Jake Kovalic, Irina Espejo Morales +28
Biological function emerges from coupled constraints across sequence, structure, regulation, evolution, and cellular context, yet most foundation models in biology are trained with…
Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
Bhavya Vasudeva, Puneesh Deora, Alberto Bietti +2
Transformer-based language models excel at in-context learning (ICL), where they can adapt to new tasks based on contextual examples, without parameter updates. In a specific form…
Learning to Recall with Transformers Beyond Orthogonal Embeddings
Nuri Mert Vural, Alberto Bietti, Mahdi Soltanolkotabi +1
Modern large language models (LLMs) excel at tasks that require storing and retrieving knowledge, such as factual recall and question answering. Transformers are central to this ca…
Representation Learning for Spatiotemporal Physical Systems
Helen Qu, Rudy Morel, Michael McCabe +4
Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evol…
Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents
Jacopo Teneggi, S. M. Bargeen A. Turzo, Tanya Marwah +4
Large language models (LLMs) are capable of emulating reasoning and using tools, creating opportunities for autonomous agents that execute complex scientific tasks. Protein design…
Understanding the Mechanisms of Fast Hyperparameter Transfer
Nikhil Ghosh, Denny Wu, Alberto Bietti
The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparame…
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
Ryotaro Kawata, Yujin Song, Alberto Bietti +4
Transformers can implement both generalizable algorithms (e.g., induction heads) and simple positional shortcuts (e.g., memorizing fixed output positions). In this work, we study h…
Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model
Rio Alexa Fear, Payel Mukhopadhyay, Michael McCabe +2
Recent advances in mechanistic interpretability have revealed that large language models (LLMs) develop internal representations corresponding not only to concrete entities but als…
Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme
Rudy Morel, Francesco Pio Ramunno, Jeff Shen +18
Conditional diffusion models provide a natural framework for probabilistic prediction of dynamical systems and have been successfully applied to fluid dynamics and weather predicti…
Universal Spectral Tokenization via Self-Supervised Panchromatic Representation Learning
Jeff Shen, Francois Lanusse, Liam Holden Parker +24
Sequential scientific data span many resolutions and domains, and unifying them into a common representation is a key step toward developing foundation models for the sciences. Ast…
Walrus: A Cross-Domain Foundation Model for Continuum Dynamics
Michael McCabe, Payel Mukhopadhyay, Tanya Marwah +22
Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unst…
AION-1: Omnimodal Foundation Model for Astronomical Sciences
Liam Parker, Francois Lanusse, Jeff Shen +24
While foundation models have shown promise across a variety of fields, astronomy still lacks a unified framework for joint modeling across its highly diverse data modalities. In th…
Aristotle: IMO-level Automated Theorem Proving
Tudor Achim, Alex Best, Alberto Bietti +20
We introduce Aristotle, an AI system that combines formal verification with informal reasoning, achieving gold-medal-equivalent performance on the 2025 International Mathematical O…
GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments
Hanlin Zhu, Tianyu Guo, Song Mei +4
As LLMs are increasingly deployed as agents, agentic reasoning - the ability to combine tool use, especially search, and reasoning - becomes a critical skill. However, it is hard t…
Learning Compositional Functions with Transformers from Easy-to-Hard Data
Zixuan Wang, Eshaan Nichani, Alberto Bietti +4
Transformer-based language models have demonstrated impressive capabilities across a range of complex reasoning tasks. Prior theoretical work exploring the expressive power of tran…
BAnG: Bidirectional Anchored Generation for Conditional RNA Design
Roman Klypa, Alberto Bietti, Sergei Grudinin
Designing RNA molecules that interact with specific proteins is a critical challenge in experimental and computational biology. Existing computational approaches require a substant…
In-context denoising with one-layer transformers: connections between attention and associative memory retrieval
Matthew Smart, Alberto Bietti, Anirvan M. Sengupta
We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfie…
Understanding Factual Recall in Transformers via Associative Memories
Eshaan Nichani, Jason D. Lee, Alberto Bietti
Large language models have demonstrated an impressive ability to perform factual recall. Prior work has found that transformers trained on factual recall tasks can store informatio…
Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task
Siavash Golkar, Alberto Bietti, Mariel Pettee +12
Transformers have revolutionized machine learning across diverse domains, yet understanding their behavior remains crucial, particularly in high-stakes applications. This paper int…
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Frederik Kunstner, Robin Yadav, Alan Milligan +2
Adam has been shown to outperform gradient descent on large language models by a larger margin than on other tasks, but it is unclear why. We show that a key factor in this perform…