1 citations · 2 across the 7 of their papers we have counts for
7 papers · 1 filter
Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks
Cheng Xu, Pengxiao Lin, Zhangchen Zhou +1
Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting…
Focus and Dilution: The Multi-stage Learning Process of Attention
Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1
Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identif…
Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge
Pengxiao Lin, Zheng-An Chen, Zhi-Qin John Xu
Large Language Models (LLMs) excel at multi-hop reasoning in distribution, yet fail on unseen compositions, a phenomenon known as the curse of two-hop reasoning. In this work, we a…
Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
Tianyi Chen, Pengxiao Lin, Zhiwei Wang +1
State Space Models (SSMs) have emerged as promising alternatives to attention mechanisms, with the Mamba architecture demonstrating impressive performance and linear complexity for…
Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Liangkai Hang, Junjie Yao, Zhiwei Bai +17
The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their…
MM: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations need
Aoming Liang, Zhaoyang Mu, Pengxiao Lin +5
Learning the evolutionary dynamics of Partial Differential Equations (PDEs) is critical in understanding dynamic systems, yet current methods insufficiently learn their representat…
Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang +2
Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the me…