16 citations · 27 across the 25 of their papers we have counts for
12 papers · 1 filter
Understanding the Mechanisms of Fast Hyperparameter Transfer
Nikhil Ghosh, Denny Wu, Alberto Bietti
The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparame…
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
Ryotaro Kawata, Yujin Song, Alberto Bietti +4
Transformers can implement both generalizable algorithms (e.g., induction heads) and simple positional shortcuts (e.g., memorizing fixed output positions). In this work, we study h…
Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model
Rio Alexa Fear, Payel Mukhopadhyay, Michael McCabe +2
Recent advances in mechanistic interpretability have revealed that large language models (LLMs) develop internal representations corresponding not only to concrete entities but als…
Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme
Rudy Morel, Francesco Pio Ramunno, Jeff Shen +18
Conditional diffusion models provide a natural framework for probabilistic prediction of dynamical systems and have been successfully applied to fluid dynamics and weather predicti…
Universal Spectral Tokenization via Self-Supervised Panchromatic Representation Learning
Jeff Shen, Francois Lanusse, Liam Holden Parker +24
Sequential scientific data span many resolutions and domains, and unifying them into a common representation is a key step toward developing foundation models for the sciences. Ast…
Walrus: A Cross-Domain Foundation Model for Continuum Dynamics
Michael McCabe, Payel Mukhopadhyay, Tanya Marwah +22
Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unst…
AION-1: Omnimodal Foundation Model for Astronomical Sciences
Liam Parker, Francois Lanusse, Jeff Shen +24
While foundation models have shown promise across a variety of fields, astronomy still lacks a unified framework for joint modeling across its highly diverse data modalities. In th…
Aristotle: IMO-level Automated Theorem Proving
Tudor Achim, Alex Best, Alberto Bietti +20
We introduce Aristotle, an AI system that combines formal verification with informal reasoning, achieving gold-medal-equivalent performance on the 2025 International Mathematical O…
GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments
Hanlin Zhu, Tianyu Guo, Song Mei +4
As LLMs are increasingly deployed as agents, agentic reasoning - the ability to combine tool use, especially search, and reasoning - becomes a critical skill. However, it is hard t…
Learning Compositional Functions with Transformers from Easy-to-Hard Data
Zixuan Wang, Eshaan Nichani, Alberto Bietti +4
Transformer-based language models have demonstrated impressive capabilities across a range of complex reasoning tasks. Prior theoretical work exploring the expressive power of tran…
BAnG: Bidirectional Anchored Generation for Conditional RNA Design
Roman Klypa, Alberto Bietti, Sergei Grudinin
Designing RNA molecules that interact with specific proteins is a critical challenge in experimental and computational biology. Existing computational approaches require a substant…
In-context denoising with one-layer transformers: connections between attention and associative memory retrieval
Matthew Smart, Alberto Bietti, Anirvan M. Sengupta
We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfie…