540 citations · 841 across the 27 of their papers we have counts for
28 papers
Focus and Dilution: The Multi-stage Learning Process of Attention
Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1
Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identif…
Optimistic Estimate Uncovers the Potential of Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible s…
Stochastic Modified Equations and Dynamics of Dropout Algorithm
Zhongwang Zhang, Yuqing Li, Tao Luo +1
Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and its impact on achieving good generalization abi…
Phase Diagram of Initial Condensation for Two-layer Neural Networks
Zhengan Chen, Yuqing Li, Tao Luo +2
The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the ea…
Linear Stability Hypothesis and Rank Stratification for Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterizat…
Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
Zhiwei Bai, Tao Luo, Zhi-Qin John Xu +1
Understanding the relation between deep and shallow neural networks is extremely important for the theoretical study of deep learning. In this work, we discover an embedding princi…
An Experimental Comparison Between Temporal Difference and Residual Gradient with Neural Network Approximation
Shuyu Yin, Tao Luo, Peilin Liu +1
Gradient descent or its variants are popular in training neural networks. However, in deep Q-learning with neural network approximation, a type of reinforcement learning, gradient…
Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin +3
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs…
Limitation of Characterizing Implicit Regularization by Data-independent Functions
Leyang Zhang, Zhi-Qin John Xu, Tao Luo +1
In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself no…
Overview frequency principle/spectral bias in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo
Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights o…
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
Yaoyu Zhang, Yuqing Li, Zhongwang Zhang +2
We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of…
MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
Lulu Zhang, Tao Luo, Yaoyu Zhang +3
In this paper, we propose a a machine learning approach via model-operator-data network (MOD-Net) for solving PDEs. A MOD-Net is driven by a model to solve PDEs based on operator r…
Embedding Principle of Loss Landscape of Deep Neural Networks
Yaoyu Zhang, Zhongwang Zhang, Tao Luo +1
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "…
Towards Understanding the Condensation of Neural Networks at Initial Training
Hanxu Zhou, Qixuan Zhou, Tao Luo +2
Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from…
An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
Tao Luo, Zheng Ma, Zhiwei Wang +2
Deep neural network (DNN) usually learns the target function from low to high frequency, which is called frequency principle or spectral bias. This frequency principle sheds light…
Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks
Yuqing Li, Tao Luo, Chao Ma
In an attempt to better understand structural benefits and generalization power of deep neural networks, we firstly present a novel graph theoretical formulation of neural network…
Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
Yaoyu Zhang, Tao Luo, Zheng Ma +1
Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to expla…
Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
Tao Luo, Zheng Ma, Zhiwei Wang +2
A supervised learning problem is to find a function in a hypothesis function space given values on isolated data points. Inspired by the frequency principle in neural networks, we…
On the exact computation of linear frequency principle dynamics and its generalization
Tao Luo, Zheng Ma, Zhi-Qin John Xu +1
Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that deep neural networks (DNNs) fit the target function from low to high frequency during the train…
A regularized deep matrix factorized model of matrix completion for image restoration
Zhemin Li, Zhi-Qin John Xu, Tao Luo +1
It has been an important approach of using matrix completion to perform image restoration. Most previous works on matrix completion focus on the low-rank property by imposing expli…
Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
Yuqing Li, Tao Luo, Nung Kwan Yip
Gradient descent yields zero training loss in polynomial time for deep neural networks despite non-convex nature of the objective function. The behavior of network in the infinite…
Phase diagram for two-layer ReLU neural networks at infinite-width limit
Tao Luo, Zhi-Qin John Xu, Zheng Ma +1
How neural network behaves during the training over different choices of hyperparameters is an important question in the study of neural networks. In this work, inspired by the pha…
Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
Tao Luo, Haizhao Yang
The problem of solving partial differential equations (PDEs) can be formulated into a least-squares minimization problem, where neural networks are used to parametrize PDE solution…
Theory of the Frequency Principle for General Deep Neural Networks
Tao Luo, Zheng Ma, Zhi-Qin John Xu +1
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Princip…