10 papers · 1 filter
Emergent Inverse-Depth Scaling From Nonlinearity In Attention
Zirui Peng, Yizhou Liu, Ziming Liu +1
Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the pa…
Optimizer-dependent training dynamics converge to the same one-third optimal data scaling
Hyunseok Lee, Mihir Basil, Yizhou Liu +1
Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a exponent emerges from learning peaked…
Emergent phases of superposition: from partial to full representation
Lihao Guo, Yizhou Liu, Jeff Gore
Large language models are thought to represent features by vectors in a hidden space of dimension given by the model's width. Superposition, in which more features are represented…
Emergent One-Third Scaling Law as Attention Tries to Concentrate
Yizhou Liu, Sara Kangaslahti, Jeff Gore
The neural scaling law relating longer training to better performance through a power law is central to today's large language models (LLMs), yet its origin remains debated. One re…
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Yizhou Liu, Jeff Gore
Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power la…
Everything at Every Scale: Scale-Invariant Diffusion with Continuous Super-Resolution
Zixin Jessie Chen, Zhuo Chen, Archer Wang +4
Creating images from noise is image generation; reconstructing fine details from coarse inputs is super-resolution. Despite their practical differences, both can be understood as r…
Inverse Depth Scaling From Most Layers Being Similar
Yizhou Liu, Sara Kangaslahti, Ziming Liu +1
Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here,…
Universal One-third Time Scaling in Learning Peaked Distributions
Yizhou Liu, Ziming Liu, Cengiz Pehlevan +1
Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic a…
Superposition unifies power-law training dynamics
Zixin Jessie Chen, Hao Chen, Yizhou Liu +1
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training…
The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective
Qiyao Liang, Jinyeop Song, Yizhou Liu +4
Weight-perturbation evolution strategies (ES) can fine-tune billion-parameter language models with surprisingly small populations (e.g., ), contradicting classical…