1 citations · 1 across the 7 of their papers we have counts for
4 papers · 1 filter
On the Nonlinearity of Learning Rate Scaling for LLM Training
Zaiwen Yang, Huaqing Zhang, Jing Xu +1
Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Exist…
Understanding Nonlinear Implicit Bias via Region Counts in Input Space
Jingwei Li, Jing Xu, Zifan Wang +2
One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little…
Scalable Model Merging with Progressive Layer-wise Distillation
Jing Xu, Jiazheng Li, Jingzhao Zhang
Model merging offers an effective way to integrate the capabilities of multiple fine-tuned models. However, the performance degradation of the merged model remains a challenge, par…
Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning
Jing Xu, Jingzhao Zhang
Fine-tuning large language models (LLM) can be costly. Parameter-efficient fine-tuning (PEFT) addresses the problems by training a fraction of the parameters, whose success reveals…