15 citations · 33 across the 7 of their papers we have counts for
7 papers
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang +4
Computation in a typical Transformer-based large language model (LLM) can be characterized by batch size, hidden dimension, number of layers, and sequence length. Until now, system…
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Guanhua Wang, Heyang Qin, Sam Ade Jacobs +6
Zero Redundancy Optimizer (ZeRO) has been used to train a wide range of large language models on massive GPUs clusters due to its ease of use, efficiency, and good scalability. How…
Learning Interpretable Models Through Multi-Objective Neural Architecture Search
Zachariah Carmichael, Tim Moon, Sam Ade Jacobs
Monumental advances in deep learning have led to unprecedented achievements across various domains. While the performance of deep neural networks is indubitable, the architectural…
Enabling Machine Learning-Ready HPC Ensembles with Merlin
J. Luc Peterson, Ben Bay, Joe Koning +17
With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensembl…
Parallelizing Training of Deep Generative Models on Massive Scientific Datasets
Sam Ade Jacobs, Brian Van Essen, David Hysom +11
Training deep neural networks on large scientific data is a challenging task that requires enormous compute power, especially if no pre-trained models exist to initialize the proce…
Scalable Topological Data Analysis and Visualization for Evaluating Data-Driven Models in Scientific Applications
Shusen Liu, Di Wang, Dan Maljovec +13
With the rapid adoption of machine learning techniques for large-scale applications in science and engineering comes the convergence of two grand challenges in visualization. First…
Distinguishing between Normal and Cancer Cells Using Autoencoder Node Saliency
Ya Ju Fan, Jonathan E. Allen, Sam Ade Jacobs +1
Gene expression profiles have been widely used to characterize patterns of cellular responses to diseases. As data becomes available, scalable learning toolkits become essential to…