5 citations · 11 across the 20 of their papers we have counts for
3 papers · 1 filter
Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project
Carolin Penke, Chelsea Maria John, Jan Ebert +2
The training of large language models (LLMs) requires substantial computational resources, complex software stacks, and carefully designed workflows to achieve scalability and effi…
Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
Jiangtao Wang, Jan Ebert, Oleg Filatov +1
Transformer models have revolutionized a wide spectrum of disciplines, especially in language processing. The recent success has proven that model size scalability is crucial for a…
JUWELS Booster -- A Supercomputer for Large-Scale AI Research
Stefan Kesselheim, Andreas Herten, Kai Krajsek +15
In this article, we present JUWELS Booster, a recently commissioned high-performance computing system at the Jülich Supercomputing Center. With its system architecture, most import…