54 citations · 59 across the 11 of their papers we have counts for
Showing 2023Show all
3 papers · 1 filter
cs.CL2023★ 3 cited
LLM360: Towards Fully Transparent Open-Source LLMs
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger +25
The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs…
cs.LG2023
FedNAR: Federated Optimization with Normalized Annealing Regularization
Junbo Li, Ang Li, Chong Tian +3
Weight decay is a standard technique to improve generalization performance in modern deep neural network optimization, and is also widely adopted in federated learning (FL) to prev…
cs.CL2023★ 2 cited
SlimPajama-DC: Understanding Data Combinations for LLM Training
Zhiqiang Shen, Tianhua Tao, Liqun Ma +8
This paper aims to understand the impacts of various data combinations (e.g., web text, Wikipedia, GitHub, books) on the pretraining of large language models using SlimPajama. Slim…