7 papers
MRNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
Mayank Mishra, Shawn Tan, Ion Stoica +2
Transformers are highly parallel but are limited to computations in the TC complexity class, excluding tasks such as entity tracking and code execution that provably require gr…
VSA: Faster Video Diffusion with Trainable Sparse Attention
Peiyuan Zhang, Yongqi Chen, Haofeng Huang +5
Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions. We turn…
Efficient Long-context Language Model Training by Core Attention Disaggregation
Yonghao Zhuang, Junda Chen, Bo Pang +6
We present core attention disaggregation (CAD), a technique that improves long-context large language model training by decoupling the core attention computation, softmax(QK^T)V, f…
Fast Video Generation with Sliding Tile Attention
Peiyuan Zhang, Yongqi Chen, Runlong Su +4
Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video…
lmgame-Bench: How Good are LLMs at Playing Games?
Lanxiang Hu, Mingjia Huo, Yuxuan Zhang +6
Playing video games requires perception, memory, and planning, exactly the faculties modern large language model (LLM) agents are expected to master. We study the major challenges…
Efficiently Scaling LLM Reasoning with Certaindex
Yichao Fu, Junda Chen, Siqi Zhu +8
Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy.…