8 citations · 10 across the 5 of their papers we have counts for
6 papers
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Mostafa Elhoushi, Alex Pretko, Nolan Dey +6
Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformer…
Knowledge Distillation of a Protein Language Model Yields a Foundational Implicit Solvent Model
Justin Airas, Bin Zhang
Implicit solvent models (ISMs) promise to deliver the accuracy of explicit solvent simulations at a fraction of the computational cost. However, despite decades of development, the…
Scaling with Collapse: Efficient and Predictable Training of LLM Families
Shane Bergsma, Bin Claire Zhang, Nolan Dey +3
Effective LLM training depends on predictable scaling of key quantities -- such as final loss and optimal hyperparameters -- with model and dataset size. Qiu et al. (2025) recently…
Don't be lazy: CompleteP enables compute-efficient deep transformers
Nolan Dey, Bin Claire Zhang, Lorenzo Noci +6
We study compute efficiency of LLM training when using different parameterizations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some…
LangYa: Revolutionizing Cross-Spatiotemporal Ocean Forecasting
Nan Yang, Chong Wang, Meihua Zhao +5
Ocean forecasting is crucial for both scientific research and societal benefits. Currently, the most accurate forecasting systems are global ocean forecasting systems (GOFSs), whic…
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang +89
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences. This could herald a new era of scient…