5 citations · 5 across the 4 of their papers we have counts for
5 papers
Relative Generalization Invariance of LLM Pretraining
Fengzhuo Zhang, Shuche Wang, Shenggui Li +7
Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. Howev…
Muon Learns More Robust and Transferable Features than Adam
Tianyu Ruan, Fengzhuo Zhang, Shuche Wang +1
Muon has recently emerged as a state-of-the-art optimizer for pretraining Large Language Models (LLMs) and vision classifiers. Despite its efficiency advantage over Adam and SGD, t…
Graph Generation with Spectral Geodesic Flow Matching
Xikun Huang, Tianyu Ruan, Chihao Zhang +1
Graph generation is a fundamental task with wide applications in modeling complex systems. Although existing methods align the spectrum or degree profile of the target graph, they…
Feature Dynamics as Implicit Data Augmentation: A Depth-Decomposed View on Deep Neural Network Generalization
Tianyu Ruan, Kuo Gai, Shihua Zhang
Why do deep networks generalize well? In contrast to classical generalization theory, we approach this fundamental question by examining not only inputs and outputs, but the evolut…
Towards understanding how attention mechanism works in deep learning
Tianyu Ruan, Shihua Zhang
Attention mechanism has been extensively integrated within mainstream neural network architectures, such as Transformers and graph attention networks. Yet, its underlying working p…