5 papers
SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
Shanghao Liu, Renze Chen, Size Zheng +4
Video diffusion transformers (vDiTs) generate high quality but pay quadratic self-attention cost, making inference prohibitive at video-token scales. The challenge is input-adaptiv…
TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference
Zhuoran Li, Zhuohang Bian, Zihao Huang +5
Large language model (LLM) serving is now limited by the key-value (KV) cache. During decode, each new token rereads prior KV state, so attention becomes a bandwidth- and capacity-…
DITRON: Distributed Multi-level Tiling Compiler for Parallel Tensor Programs
Size Zheng, Xuegui Zheng, Hanshi Sun +16
The scaling of large language models (LLMs) is currently bottlenecked by the rigidity of distributed programming. While high-performance libraries like CuBLAS and NCCL provide opti…
DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation
Zihao Zheng, Xiuping Cui, Size Zheng +4
As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization me…
Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
Size Zheng, Wenlei Bao, Qi Hou +19
In this report, we propose Triton-distributed, an extension of existing Triton compiler, to overcome the programming challenges in distributed AI systems. Triton-distributed is the…