70 citations · 164 across the 35 of their papers we have counts for
4 papers · 2 filters
Optimistic Estimate Uncovers the Potential of Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible s…
Stochastic Modified Equations and Dynamics of Dropout Algorithm
Zhongwang Zhang, Yuqing Li, Tao Luo +1
Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and its impact on achieving good generalization abi…
Loss Spike in Training Neural Networks
Xiaolong Li, Zhi-Qin John Xu, Zhongwang Zhang
In this work, we investigate the mechanism underlying loss spikes observed during neural network training. When the training enters a region with a lower-loss-as-sharper (LLAS) str…
Phase Diagram of Initial Condensation for Two-layer Neural Networks
Zhengan Chen, Yuqing Li, Tao Luo +2
The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the ea…