1 citations · 1 across the 8 of their papers we have counts for
11 papers
Understanding the Mechanisms of Fast Hyperparameter Transfer
Nikhil Ghosh, Denny Wu, Alberto Bietti
The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparame…
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
Ryotaro Kawata, Yujin Song, Alberto Bietti +4
Transformers can implement both generalizable algorithms (e.g., induction heads) and simple positional shortcuts (e.g., memorizing fixed output positions). In this work, we study h…
Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model
Rio Alexa Fear, Payel Mukhopadhyay, Michael McCabe +2
Recent advances in mechanistic interpretability have revealed that large language models (LLMs) develop internal representations corresponding not only to concrete entities but als…
Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme
Rudy Morel, Francesco Pio Ramunno, Jeff Shen +18
Conditional diffusion models provide a natural framework for probabilistic prediction of dynamical systems and have been successfully applied to fluid dynamics and weather predicti…
Universal Spectral Tokenization via Self-Supervised Panchromatic Representation Learning
Jeff Shen, Francois Lanusse, Liam Holden Parker +24
Sequential scientific data span many resolutions and domains, and unifying them into a common representation is a key step toward developing foundation models for the sciences. Ast…
AION-1: Omnimodal Foundation Model for Astronomical Sciences
Liam Parker, Francois Lanusse, Jeff Shen +24
While foundation models have shown promise across a variety of fields, astronomy still lacks a unified framework for joint modeling across its highly diverse data modalities. In th…