163 citations · 165 across the 6 of their papers we have counts for
3 papers · 1 filter
Cross-Layer Energy Analysis of Multimodal Training on Grace Hopper Superchips
Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase +4
Multimodal deep learning models enable joint learning across heterogeneous data sources, including text, images, and video, but their rapid scaling introduces significant memory an…
Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer
Jinghan Yao, Sam Ade Jacobs, Masahiro Tanaka +3
Large Language Models (LLMs) with long context capabilities are integral to complex tasks in natural language processing and computational biology, such as text generation and prot…
Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelis
Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko +4
Deep neural network (DNN) training continues to scale rapidly in terms of model size, data volume, and sequence length, to the point where multiple machines are required to fit lar…