6 papers · 1 filter
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
Han Wang, Jintao Zhang, Kai Jiang +3
LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capability break down, and why? We p…
SageBwd: A Trainable Low-bit Attention
Jintao Zhang, Marco Chen, Haoxu Wang +5
Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In pri…
DiffusionNFT: Online Diffusion Reinforcement with Forward Process
Kaiwen Zheng, Huayu Chen, Haotian Ye +7
Online reinforcement learning (RL) has been central to post-training language models, but its extension to diffusion models remains challenging due to intractable likelihoods. Rece…
SLA2: Sparse-Linear Attention with Learnable Routing and QAT
Jintao Zhang, Haoxu Wang, Kai Jiang +6
Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a…
SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
Jintao Zhang, Jia Wei, Pengle Zhang +6
The efficiency of attention is important due to its quadratic time complexity. We enhance the efficiency of attention through two key contributions: First, we leverage the new FP4…
SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
Jintao Zhang, Haoxu Wang, Kai Jiang +10
In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck due to the long sequence length and the quadratic complexity. We f…