6 papers
Relative Generalization Invariance of LLM Pretraining
Fengzhuo Zhang, Shuche Wang, Shenggui Li +7
Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. Howev…
Neural Networks Provably Learn Spectral Representations for Group Composition
Jianliang He, Leda Wang, Fengzhuo Zhang +2
Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group co…
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
Jianliang He, Leda Wang, Siyu Chen +1
We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the l…
In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
Jianliang He, Xintian Pan, Siyu Chen +1
We study how multi-head softmax attention models are trained to perform in-context learning on linear data. Through extensive empirical experiments and rigorous theoretical analysi…
From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems
Jianliang He, Siyu Chen, Fengzhuo Zhang +1
In this work, from a theoretical lens, we aim to understand why large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. To thi…
Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation
Jianliang He, Han Zhong, Zhuoran Yang
We study infinite-horizon average-reward Markov decision processes (AMDPs) in the context of general function approximation. Specifically, we propose a novel algorithmic framework…