15 citations · 16 across the 10 of their papers we have counts for
5 papers · 1 filter
From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers
Ziming Liu, Sophia Sanborn, Surya Ganguli +1
Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions th…
Neural Thermodynamic Laws for Large Language Model Training
Ziming Liu, Yizhou Liu, Jeff Gore +1
Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fres…
Physics of Skill Learning
Ziming Liu, Yizhou Liu, Eric J. Michaud +2
We aim to understand physics of skill learning, i.e., how skills are learned in neural networks during training. We start by observing the Domino effect, i.e., skills are learned s…
FOCUS: First Order Concentrated Updating Scheme
Yizhou Liu, Ziming Liu, Jeff Gore
Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further. Based on the do…
Fokker-Planck to Callan-Symanzik: evolution of weight matrices under training
Wei Bu, Uri Kol, Ziming Liu
The dynamical evolution of a neural network during training has been an incredibly fascinating subject of study. First principal derivation of generic evolution of variables in sta…