RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

Zekun Zhang, Yixiang Cai, Yuxi Liu +9

Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequence length and quickly becomes t…