8 citations · 8 across the 3 of their papers we have counts for
3 papers
Cross-Layer Energy Analysis of Multimodal Training on Grace Hopper Superchips
Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase +4
Multimodal deep learning models enable joint learning across heterogeneous data sources, including text, images, and video, but their rapid scaling introduces significant memory an…
MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation
Jinghan Yao, Sam Adé Jacobs, Walid Krichene +2
Long-context decoding in LLMs is IO-bound: each token re-reads an ever-growing KV cache. Prior accelerations cut bytes via compression, which lowers fidelity, or selection/eviction…
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang +4
Computation in a typical Transformer-based large language model (LLM) can be characterized by batch size, hidden dimension, number of layers, and sequence length. Until now, system…