7 papers
Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Jiajie Zhao, Jianxing Wang, Junjie Yang +2
We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization. Extending Theorem 1 from Pesme & Flammarion (2023), we gene…
Towards Understanding Adam Convergence on Highly Degenerate Polynomials
Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou +2
Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike pri…
Adaptive Preconditioners Trigger Loss Spikes in Adam
Zhiwei Bai, Zhangchen Zhou, Jiajie Zhao +6
Loss spikes commonly emerge during neural network training with the Adam optimizer across diverse architectures and scales, yet their underlying mechanism remains elusive. While pr…
Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Liangkai Hang, Junjie Yao, Zhiwei Bai +17
The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their…
Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization
Yaoyu Zhang, Leyang Zhang, Zhongwang Zhang +1
Determining whether deep neural network (DNN) models can reliably recover target functions at overparameterization is a critical yet complex issue in the theory of deep learning. T…
Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Target
Jiajie Zhao, Zhiwei Bai, Yaoyu Zhang
Overparameterized models like deep neural networks have the intriguing ability to recover target functions with fewer sampled data points than parameters (see arXiv:2307.08921). To…