VideoSEMA: a scalable and efficient Mamba-like attention for video understanding
arXiv:2607.14711
The paper introduces VideoSEMA, a split space‑time attention model for video classification that combines a scalable Mamba‑like spatial attention block with softmax temporal attention, achieving strong accuracy and efficiency on Kinetics‑400 and Something‑Something V2 benchmarks.
Abstract
We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard to on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.
15 pages, 3 figures