4 citations · 6 across the 8 of their papers we have counts for
4 papers · 1 filter
Optimal Scaling Needs Optimal Norm
Oleg Filatov, Jiangtao Wang, Jan Ebert +1
Despite recent progress in optimal hyperparameter transfer under model and dataset scaling, no unifying explanatory principle has been established. For Adam and Scion optimizers, w…
Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project
Carolin Penke, Chelsea Maria John, Jan Ebert +2
The training of large language models (LLMs) requires substantial computational resources, complex software stacks, and carefully designed workflows to achieve scalability and effi…
Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
Jiangtao Wang, Jan Ebert, Oleg Filatov +1
Transformer models have revolutionized a wide spectrum of disciplines, especially in language processing. The recent success has proven that model size scalability is crucial for a…
The Artificial Scientist -- in-transit Machine Learning of Plasma Simulations
Jeffrey Kelling, Vicente Bolea, Michael Bussmann +19
Increasing HPC cluster sizes and large-scale simulations that produce petabytes of data per run, create massive IO and storage challenges for analysis. Deep learning-based techniqu…