540 citations · 833 across the 33 of their papers we have counts for
24 papers · 1 filter
An overview of condensation phenomenon in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Zhangchen Zhou
In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural netwo…
A rationale from frequency perspective for grokking in training neural network
Zhangchen Zhou, Yaoyu Zhang, Zhi-Qin John Xu
Grokking is the phenomenon where neural networks NNs initially fit the training data and later generalize to the test data during training. In this paper, we empirically provide a…
Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang +2
Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the me…
Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization
Leyang Zhang, Yaoyu Zhang, Tao Luo
Under mild assumptions, we investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. Utilizing novel techniques, we demonstrate…
Optimistic Estimate Uncovers the Potential of Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible s…
Linear Stability Hypothesis and Rank Stratification for Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterizat…
Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
Zhiwei Bai, Tao Luo, Zhi-Qin John Xu +1
Understanding the relation between deep and shallow neural networks is extremely important for the theoretical study of deep learning. In this work, we discover an embedding princi…
Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin +3
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs…
Limitation of Characterizing Implicit Regularization by Data-independent Functions
Leyang Zhang, Zhi-Qin John Xu, Tao Luo +1
In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself no…
Overview frequency principle/spectral bias in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo
Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights o…
A deep learning-based model reduction (DeePMR) method for simplifying chemical kinetics
Zhiwei Wang, Yaoyu Zhang, Enhan Zhao +4
A deep learning-based model reduction (DeePMR) method for simplifying chemical kinetics is proposed and validated using high-temperature auto-ignitions, perfectly stirred reactors…
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
Yaoyu Zhang, Yuqing Li, Zhongwang Zhang +2
We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of…
Embedding Principle of Loss Landscape of Deep Neural Networks
Yaoyu Zhang, Zhongwang Zhang, Tao Luo +1
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "…
Towards Understanding the Condensation of Neural Networks at Initial Training
Hanxu Zhou, Qixuan Zhou, Tao Luo +2
Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from…
An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
Tao Luo, Zheng Ma, Zhiwei Wang +2
Deep neural network (DNN) usually learns the target function from low to high frequency, which is called frequency principle or spectral bias. This frequency principle sheds light…
Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
Yaoyu Zhang, Tao Luo, Zheng Ma +1
Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to expla…
On the exact computation of linear frequency principle dynamics and its generalization
Tao Luo, Zheng Ma, Zhi-Qin John Xu +1
Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that deep neural networks (DNNs) fit the target function from low to high frequency during the train…
Phase diagram for two-layer ReLU neural networks at infinite-width limit
Tao Luo, Zhi-Qin John Xu, Zheng Ma +1
How neural network behaves during the training over different choices of hyperparameters is an important question in the study of neural networks. In this work, inspired by the pha…
A priori generalization error for two-layer ReLU neural network through minimum norm solution
Zhi-Qin John Xu, Jiwei Zhang, Yaoyu Zhang +1
We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the…
Theory of the Frequency Principle for General Deep Neural Networks
Tao Luo, Zheng Ma, Zhi-Qin John Xu +1
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Princip…
Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo +1
It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit…
A type of generalization error induced by initialization in deep neural networks
Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo +1
How initialization and loss function affect the learning of a deep neural network (DNN), specifically its generalization error, is an important problem in practice. In this work, b…
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo +2
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often…
Training behavior of deep neural network in frequency domain
Zhi-Qin John Xu, Yaoyu Zhang, Yanyang Xiao
Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the stud…