11 papers
When and How Does Muon Precondition? The Role of Structured Gradient Noise
Xiaoran Chen, Binghui Li, Liwei Wang +1
Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry…
Are Parameter-Efficient Fine-tuning Methods Really Different?
Yikuan Li, Pinyan Lu, Fanghui Liu
Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and dif…
Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise
Linyu Liu, Pinyan Lu
Highly over-parameterized models can simultaneously memorize noisy labels and generalize well, yet how these behaviors coexist remains poorly understood. In this work, we investiga…
SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoning
Zhengyang Ai, Zikang Shan, Xiaodong Ai +3
Process supervision has emerged as a promising approach for enhancing LLM reasoning, yet existing methods fail to distinguish meaningful progress from mere verbosity, leading to li…
Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Use
Hanbing Liu, Chunhao Tian, Nan An +4
We study budget-constrained tool-augmented agents, where a large language model must solve multi-step tasks by invoking external tools under a strict monetary budget. We formalize…
BAPS: A Fine-Grained Low-Precision Scheme for Softmax in Attention via Block-Aware Precision reScaling
Zisheng Ye, Xiaoyu He, Maoyuan Song +10
As the performance gains from accelerating quantized matrix multiplication plateau, the softmax operation becomes the critical bottleneck in Transformer inference. This bottleneck…
A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention
Xiaowei Ye, Xiaoyu He, Chao Liao +2
Transformers serve as the foundation of most modern large language models. To mitigate the quadratic complexity of standard full attention, various efficient attention mechanisms,…
Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
Binghui Li, Kaifei Wang, Han Zhong +2
Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory. We study Mu…
Understanding the Ability of LLMs to Handle Character-Level Perturbation
Anyuan Zhuo, Xuefei Ning, Ningyuan Li +3
This work investigates the resilience of contemporary large language models (LLMs) against frequent character-level perturbations. We examine three types of character-level perturb…
Two-State Spin Systems with Negative Interactions
Yumou Fei, Leslie Ann Goldberg, Pinyan Lu
We study the approximability of computing the partition functions of two-state spin systems. The problem is parameterized by a symmetric matrix. Previous results on thi…
Efficient Training of Multi-task Neural Solver for Combinatorial Optimization
Chenguang Wang, Zhang-Hua Fu, Pinyan Lu +1
Efficiently training a multi-task neural solver for various combinatorial optimization problems (COPs) has been less studied so far. Naive application of conventional multi-task le…