From the 1 of 6 linked papers with an AI index.
6 papers
Towards joint scaling laws with optimal batch size schedules
Jiaxiang Li, Zhiqi Bu, Shiyun Xu
The paper derives a theoretical relationship between learning rate and batch size schedules using convex optimization, and proposes a closed‑form optimal batch size schedule that i…
Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate
Zhiqi Bu, Shiyun Xu, Jialin Mao
Deep learning has non-convex loss landscape and its optimization dynamics is hard to analyze or control. Nevertheless, the dynamics can be empirically convex-like across various ta…
FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Models
Md Kowsher, Nusrat Jahan Prottasha, Shiyun Xu +4
Bidirectional language models have better context understanding and perform better than unidirectional models on natural language understanding tasks, yet the theoretical reasons b…
Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection
Shiyun Xu, Zhiqi Bu
Parameter-efficient fine-tuning (PEFT) is a highly effective approach for adapting large pre-trained models to downstream tasks with minimal computational overhead. At the core, PE…
A Hessian-informed hyperparameter optimization for differential learning rate
Shiyun Xu, Zhiqi Bu, Yiliang Zhang +1
Differential learning rate (DLR), a technique that applies different learning rates to different model parameters, has been widely used in deep learning and achieved empirical succ…
Gradient descent with generalized Newton's method
Zhiqi Bu, Shiyun Xu
We propose the generalized Newton's method (GeN) -- a Hessian-informed approach that applies to any optimizer such as SGD and Adam, and covers the Newton-Raphson method as a sub-ca…