2 citations · 11 across the 17 of their papers we have counts for
17 papers
Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Liangkai Hang, Junjie Yao, Zhiwei Bai +17
The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their…
Reasoning Bias of Next Token Prediction Training
Pengxiao Lin, Zhongwang Zhang, Zhi-Qin John Xu
Since the inception of Large Language Models (LLMs), the quest to efficiently train them for superior reasoning capabilities has been a pivotal challenge. The dominant training par…
An Analysis for Reasoning Bias of Language Models with Small Initialization
Junjie Yao, Zhongwang Zhang, Zhi-Qin John Xu
Transformer-based Large Language Models (LLMs) have revolutionized Natural Language Processing by demonstrating exceptional performance across diverse tasks. This study investigate…
Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang +2
Transformers have demonstrated impressive capabilities across various tasks, yet their performance on compositional problems remains a subject of debate. In this study, we investig…
Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization
Yaoyu Zhang, Leyang Zhang, Zhongwang Zhang +1
Determining whether deep neural network (DNN) models can reliably recover target functions at overparameterization is a critical yet complex issue in the theory of deep learning. T…
Loss Jump During Loss Switch in Solving PDEs with Neural Networks
Zhiwei Wang, Lulu Zhang, Zhongwang Zhang +1
Using neural networks to solve partial differential equations (PDEs) is gaining popularity as an alternative approach in the scientific computing community. Neural networks can int…
Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism
Zhiwei Wang, Yunji Wang, Zhongwang Zhang +7
Large language models have consistently struggled with complex reasoning tasks, such as mathematical problem-solving. Investigating the internal reasoning mechanisms of these model…
Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang +2
Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the me…
Anchor function: a type of benchmark functions for studying language models
Zhongwang Zhang, Zhiwei Wang, Junjie Yao +4
Understanding transformer-based language models is becoming increasingly crucial, particularly as they play pivotal roles in advancing towards artificial general intelligence. Howe…
Optimistic Estimate Uncovers the Potential of Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible s…
Stochastic Modified Equations and Dynamics of Dropout Algorithm
Zhongwang Zhang, Yuqing Li, Tao Luo +1
Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and its impact on achieving good generalization abi…
Loss Spike in Training Neural Networks
Xiaolong Li, Zhi-Qin John Xu, Zhongwang Zhang
In this work, we investigate the mechanism underlying loss spikes observed during neural network training. When the training enters a region with a lower-loss-as-sharper (LLAS) str…
Linear Stability Hypothesis and Rank Stratification for Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang +3
Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterizat…
Implicit regularization of dropout
Zhongwang Zhang, Zhi-Qin John Xu
It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we presen…
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
Yaoyu Zhang, Yuqing Li, Zhongwang Zhang +2
We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of…
Dropout in Training Neural Networks: Flatness of Solution and Noise Structure
Zhongwang Zhang, Hanxu Zhou, Zhi-Qin John Xu
It is important to understand how the popular regularization method dropout helps the neural network training find a good generalization solution. In this work, we show that the tr…
Embedding Principle of Loss Landscape of Deep Neural Networks
Yaoyu Zhang, Zhongwang Zhang, Tao Luo +1
Understanding the structure of loss landscape of deep neural networks (DNNs)is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "…