ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention

Yuhong Chou, Zehao Liu, Ruijie Zhu +6

Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-lon…