5 papers · 1 filter
Estimating the Empowerment of Language Model Agents
Jinyeop Song, Jeff Gore, Max Kleiman-Weiner
As language model (LM) agents become increasingly capable and adopted in real-world applications, there is a growing need for scalable evaluation frameworks beyond costly, manually…
Neural Thermodynamic Laws for Large Language Model Training
Ziming Liu, Yizhou Liu, Jeff Gore +1
Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fres…
Superposition Yields Robust Neural Scaling
Yizhou Liu, Ziming Liu, Jeff Gore
The success of today's large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decrease…
Physics of Skill Learning
Ziming Liu, Yizhou Liu, Eric J. Michaud +2
We aim to understand physics of skill learning, i.e., how skills are learned in neural networks during training. We start by observing the Domino effect, i.e., skills are learned s…
FOCUS: First Order Concentrated Updating Scheme
Yizhou Liu, Ziming Liu, Jeff Gore
Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further. Based on the do…