13 papers
Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
Yang Cao, Yingyu Liang, Zhenmei Shi +1
Tensor Attention, a multi-view attention that is able to capture high-order correlations among multiple modalities, can overcome the representational limitations of classical matri…
On Fine-Grained I/O Complexity of Attention Backward Passes
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +3
Large Language Models (LLMs) exhibit exceptional proficiency in handling extensive context windows in natural language. Nevertheless, the quadratic scaling of attention computation…
Provable Differentially Private Computation of the Cross-Attention Mechanism
Yekun Ke, Yingyu Liang, Zhenmei Shi +2
Cross-attention has emerged as a cornerstone module in modern artificial intelligence, underpinning critical applications such as retrieval-augmented generation (RAG), system promp…
Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs
Yufa Zhou, Shaobo Wang, Xingyu Dong +7
Directly training Large Language Models (LLMs) for Multi-Agent Systems (MAS) remains challenging due to intricate reward modeling, dynamic agent interactions, and demanding general…
FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge
Xuan Shen, Weize Ma, Yufa Zhou +11
Auto-regressive (AR) models, initially successful in language generation, have recently shown promise in visual generation tasks due to their superior sampling efficiency. Unlike i…
DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
Xuan Shen, Chenxia Han, Yufa Zhou +7
Diffusion transformer-based video generation models (DiTs) have recently attracted widespread attention for their excellent generation quality. However, their computational cost re…