SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation

Animesh Karnewar, Denis Korzhenkov, Amirhossein Habibian +1

Video Diffusion Transformers (DiTs) spend most of their compute inside the Self-Attention operation, whose cost grows quadratically, $\mathcal{O}(n^2)$, with the number of latent t…