Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Gyuryang Heo, Timothy Ngotiaoco, Kazuki Irie +2

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient tra…