13 papers
A Fine-Grained Analysis of the LoRA Fine-Tuning Landscape with Implications for Data Selection
Bowen Zhang, Changrui Fang, Xinsong Ma +2
Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning, yet a fundamental practical question remains unresolved: how should the adapter rank…
Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Zixuan Gong, Zeyu Gan, Jiaye Teng +1
Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it join…
Hyperball May Not Be a Free Lunch
Yihao Xiao, Jialong Sun, Zitian Gao +5
For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing…
Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
Tianqi Shen, Jinji Yang, Runze Shi +3
Recently, Muon has gained substantial attention as an appealing alternative to Adam-like optimizers, with many works highlighting its advantages through spectral normalization and…
Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
Huangyu Xu, Jingqin Yang, Qianqian Xu +1
Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is regularization. However, it may encounter op…
Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks
Wenquan Ma, Yang Sui, Jiaye Teng +3
Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize under non-convex…
Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better
Yizhou Min, Yizhou Lu, Lanqi Li +2
Conformal prediction(CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work criticall…
Matrix Sensing with Kernel Optimal Loss: Robustness and Optimization Landscape
Xinyuan Song, Ziye Ma
In this paper we study how the choice of loss functions of non-convex optimization problems affects their robustness and optimization landscape, through the study of noisy matrix s…
What Makes Looped Transformers Perform Better Than Non-Recursive Ones
Zixuan Gong, Yong Liu, Jiaye Teng
While looped transformers (termed as Looped-Attn) often outperform standard transformers (termed as Single-Attn) on complex reasoning tasks, the mechanism for this advantage remain…
Smoothing-Based Conformal Prediction for Balancing Efficiency and Interpretability
Mingyi Zheng, Hongyu Jiang, Yizhou Lu +1
Conformal Prediction (CP) is a distribution-free framework for constructing statistically rigorous prediction sets. While popular variants such as CD-split improve CP's efficiency,…
Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
Yifan Sun, Yushan Liang, Zhen Zhang +2
Self-improvement is a significant techniques within the realm of large language model (LLM), aiming to enhance the LLM performance without relying on external data. Despite its sig…
Minimax Optimal Two-Stage Algorithm For Moment Estimation Under Covariate Shift
Zhen Zhang, Xin Liu, Shaoli Wang +1
Covariate shift occurs when the distribution of input features differs between the training and testing phases. In covariate shift, estimating an unknown function's moment is a cla…
Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformers
Zixuan Gong, Shijia Li, Yong Liu +1
Transformers may exhibit two-stage training dynamics during the real-world training process. For instance, when training GPT-2 on the Counterfact dataset, the answers progress from…