13 citations · 29 across the 14 of their papers we have counts for
14 papers
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Yizhou Liu, Jeff Gore
Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power la…
Inverse Depth Scaling From Most Layers Being Similar
Yizhou Liu, Sara Kangaslahti, Ziming Liu +1
Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here,…
Universal One-third Time Scaling in Learning Peaked Distributions
Yizhou Liu, Ziming Liu, Cengiz Pehlevan +1
Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic a…
Superposition unifies power-law training dynamics
Zixin Jessie Chen, Hao Chen, Yizhou Liu +1
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training…
The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective
Qiyao Liang, Jinyeop Song, Yizhou Liu +4
Weight-perturbation evolution strategies (ES) can fine-tune billion-parameter language models with surprisingly small populations (e.g., ), contradicting classical…
Neural Thermodynamic Laws for Large Language Model Training
Ziming Liu, Yizhou Liu, Jeff Gore +1
Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fres…
Superposition Yields Robust Neural Scaling
Yizhou Liu, Ziming Liu, Jeff Gore
The success of today's large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decrease…
Physics of Skill Learning
Ziming Liu, Yizhou Liu, Eric J. Michaud +2
We aim to understand physics of skill learning, i.e., how skills are learned in neural networks during training. We start by observing the Domino effect, i.e., skills are learned s…
FOCUS: First Order Concentrated Updating Scheme
Yizhou Liu, Ziming Liu, Jeff Gore
Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further. Based on the do…
On the Computational Complexity of Schrödinger Operators
Yufan Zheng, Jiaqi Leng, Yizhou Liu +1
We study computational problems related to the Schrödinger operator in the real space under the condition that (i) the potential function is smooth and has its valu…
Complex fractal trainability boundary can arise from trivial non-convexity
Yizhou Liu
Training neural networks involves optimizing parameters to minimize a loss function, where the nature of the loss function and the optimization strategy are crucial for effective t…
Quantum Langevin Dynamics for Optimization
Zherui Chen, Yuchen Lu, Hao Wang +2
We initiate the study of utilizing Quantum Langevin Dynamics (QLD) to solve optimization problems, particularly those non-convex objective functions that present substantial obstac…
On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walks
Yizhou Liu, Weijie J. Su, Tongyang Li
Classical algorithms are often not effective for solving nonconvex optimization problems where local minima are separated by high barriers. In this paper, we explore possible quant…
Relating measurement disturbance, information and orthogonality
Yizhou Liu, John B. DeBrota
In the general theory of quantum measurement, one associates a positive semidefinite operator on a -dimensional Hilbert space to each of the possible outcomes of an arbitrar…