TASP: Topology-aware Sequence Parallelism

Yida Wang, Ke Hong, Xiuhong Li +4

Long-context large language models (LLMs) face constraints due to the quadratic complexity of the self-attention mechanism. The mainstream sequence parallelism (SP) method, Ring At…