JUWELS Booster -- A Supercomputer for Large-Scale AI Research

Abstract

In this article, we present JUWELS Booster, a recently commissioned high-performance computing system at the Jülich Supercomputing Center. With its system architecture, most importantly its large number of powerful Graphics Processing Units (GPUs) and its fast interconnect via InfiniBand, it is an ideal machine for large-scale Artificial Intelligence (AI) research and applications. We detail its system architecture, parallel, distributed model training, and benchmarks indicating its outstanding performance. We exemplify its potential for research application by presenting large-scale AI research highlights from various scientific fields that require such a facility.

arXiv comment12 pages, 5 figures. Accepted at ISC 2021, Workshop Deep Learning on Supercomputers. This is a duplicate submission as my previous submission is on hold for several weeks now and my attempts to contact the moderators failed

References in corpus (19)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Reconciling modern machine learning practice and the bias-variance trade-off
• Identification of direct residue contacts in protein-protein interaction by message passing
• cuDNN: Efficient Primitives for Deep Learning
• BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Hierarchical Representations for Efficient Architecture Search
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Large-scale identification of coevolution signals across homo-oligomeric protein interfaces by Direct Coupling Analysis
• 8-Bit Approximations for Parallelism in Deep Learning
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• Exascale Deep Learning for Scientific Inverse Problems
• HeAT -- a Distributed and GPU-accelerated Tensor Framework for Data Analytics
• On the Utility of Gradient Compression in Distributed Training Systems
• BigEarthNet Dataset with A New Class-Nomenclature for Remote Sensing Image Understanding