540 citations · 833 across the 33 of their papers we have counts for
33 papers
An overview of condensation phenomenon in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Zhangchen Zhou
In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural netwo…
Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang +2
Transformers have demonstrated impressive capabilities across various tasks, yet their performance on compositional problems remains a subject of debate. In this study, we investig…
A rationale from frequency perspective for grokking in training neural network
Zhangchen Zhou, Yaoyu Zhang, Zhi-Qin John Xu
Grokking is the phenomenon where neural networks NNs initially fit the training data and later generalize to the test data during training. In this paper, we empirically provide a…
Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism
Zhiwei Wang, Yunji Wang, Zhongwang Zhang +7
Large language models have consistently struggled with complex reasoning tasks, such as mathematical problem-solving. Investigating the internal reasoning mechanisms of these model…
Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang +2
Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the me…
Solving multiscale dynamical systems by deep learning
Junjie Yao, Yuxiao Yi, Liangkai Hang +5
Multiscale dynamical systems, modeled by high-dimensional stiff ordinary differential equations (ODEs) with wide-ranging characteristic timescales, arise across diverse fields of s…
Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization
Leyang Zhang, Yaoyu Zhang, Tao Luo
Under mild assumptions, we investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. Utilizing novel techniques, we demonstrate…
Optimistic Estimate Uncovers the Potential of Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible s…
Linear Stability Hypothesis and Rank Stratification for Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterizat…
Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
Zhiwei Bai, Tao Luo, Zhi-Qin John Xu +1
Understanding the relation between deep and shallow neural networks is extremely important for the theoretical study of deep learning. In this work, we discover an embedding princi…
Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin +3
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs…
Limitation of Characterizing Implicit Regularization by Data-independent Functions
Leyang Zhang, Zhi-Qin John Xu, Tao Luo +1
In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself no…
Overview frequency principle/spectral bias in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo
Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights o…
A multi-scale sampling method for accurate and robust deep neural network to predict combustion chemical kinetics
Tianhan Zhang, Yuxiao Yi, Yifan Xu +4
Machine learning has long been considered as a black box for predicting combustion chemical kinetics due to the extremely large number of parameters and the lack of evaluation stan…
A deep learning-based model reduction (DeePMR) method for simplifying chemical kinetics
Zhiwei Wang, Yaoyu Zhang, Enhan Zhao +4
A deep learning-based model reduction (DeePMR) method for simplifying chemical kinetics is proposed and validated using high-temperature auto-ignitions, perfectly stirred reactors…
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
Yaoyu Zhang, Yuqing Li, Zhongwang Zhang +2
We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of…
Data-informed Deep Optimization
Lulu Zhang, Zhi-Qin John Xu, Yaoyu Zhang
Complex design problems are common in the scientific and industrial fields. In practice, objective functions or constraints of these problems often do not have explicit formulas, a…
MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
Lulu Zhang, Tao Luo, Yaoyu Zhang +3
In this paper, we propose a a machine learning approach via model-operator-data network (MOD-Net) for solving PDEs. A MOD-Net is driven by a model to solve PDEs based on operator r…
Embedding Principle of Loss Landscape of Deep Neural Networks
Yaoyu Zhang, Zhongwang Zhang, Tao Luo +1
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "…
Towards Understanding the Condensation of Neural Networks at Initial Training
Hanxu Zhou, Qixuan Zhou, Tao Luo +2
Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from…
An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
Tao Luo, Zheng Ma, Zhiwei Wang +2
Deep neural network (DNN) usually learns the target function from low to high frequency, which is called frequency principle or spectral bias. This frequency principle sheds light…
Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
Yaoyu Zhang, Tao Luo, Zheng Ma +1
Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to expla…
Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
Tao Luo, Zheng Ma, Zhiwei Wang +2
A supervised learning problem is to find a function in a hypothesis function space given values on isolated data points. Inspired by the frequency principle in neural networks, we…
A deep learning-based ODE solver for chemical kinetics
Tianhan Zhang, Yaoyu Zhang, Weinan E +1
Developing efficient and accurate algorithms for chemistry integration is a challenging task due to its strong stiffness and high dimensionality. The current work presents a deep l…