most citedDeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

8 citations · 8 across the 1 of their papers we have counts for

collaborators

1 paper

cs.LG2023★ 8 cited

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang +4

Computation in a typical Transformer-based large language model (LLM) can be characterized by batch size, hidden dimension, number of layers, and sequence length. Until now, system…