11 citations · 16 across the 13 of their papers we have counts for
15 papers
Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows
Bochao Feng, Jianjiang Li, Haojie Wang +4
Agentic LLM workflows consist of sequences of model turns interleaved with tool interactions, so their end-to-end completion time depends not only on inference speed but also on wh…
Epoch: Compiling Diffusion Blocks for Sparse MoE Serving
Jianian Zhu, Hang Wu, Yinghui Li +3
Diffusion language models generate text by refining a fixed-size block of token positions through many forward passes, a loop that does not match the per-forward execution unit use…
RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
Heng Xu, Zhiwei Yu, Chengze Du +5
Training Mixture-of-Experts (MoE) models introduces sparse and highly imbalanced all-to-all communication that dominates iteration time. Conventional load-balancing methods fail to…
LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers
Enda Yu, Dezun Dong, Zhaoning Zhang +6
Mixture-of-Experts (MoE) models face memory and PCIe latency bottlenecks when deployed on commodity hardware. Offloading expert weights to CPU memory results in PCIe transfer laten…
NineToothed: A Triton-Based High-Level Domain-Specific Language for Machine Learning
Jiacheng Huang, Zimin Li, Yinghui Li +1
The emergence of deep learning domain-specific languages (DSLs) has substantially reduced the obstacles in developing high-performance, cross-platform compute kernels. However, cur…
Temporal-Aware GPU Resource Allocation for Distributed LLM Inference via Reinforcement Learning
Chengze Du, Zhiwei Yu, Heng Xu +3
The rapid growth of large language model (LLM) services imposes increasing demands on distributed GPU inference infrastructure. Most existing scheduling systems follow a reactive p…
SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models
Hang Wu, Jianian Zhu, Yinghui Li +3
Large Language Models (LLMs) present a critical trade-off between inference quality and computational cost: larger models offer superior capabilities but incur significant latency,…
FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
Jianian Zhu, Hang Wu, Haojie Wang +4
Multi-modal Large Language Models (MLLMs) serving systems commonly employ KV-cache compression to reduce memory footprint. However, existing compression methods introduce significa…
FlexPie: Accelerate Distributed Inference on Edge Devices with Flexible Combinatorial Optimization[Technical Report]
Runhua Zhang, Hongxu Jiang, Jinkun Geng +3
The rapid advancement of deep learning has catalyzed the development of novel IoT applications, which often deploy pre-trained deep neural network (DNN) models across multiple edge…
TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators
Jianling Li, Shangzhan Li, Zhenye Gao +9
Triton, a high-level Python-like language designed for building efficient GPU kernels, is widely adopted in deep learning frameworks due to its portability, flexibility, and access…
Optimal Kernel Orchestration for Tensor Programs with Korch
Muyan Hu, Ashwin Venkatram, Shreyashri Biswas +7
Kernel orchestration is the task of mapping the computation defined in different operators of a deep neural network (DNN) to the execution of GPU kernels on modern hardware platfor…
PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR
Zixuan Ma, Haojie Wang, Jingze Xing +7
Deep neural networks (DNNs) are of critical use in different domains. To accelerate DNN computation, tensor compilers are proposed to generate efficient code on different domain-sp…
Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices
Zhang Runhua, Jiang Hongxu, Tian Fangzheng +7
Edge computing has been emerging as a popular scenario for model inference. However, the inference performance on edge devices (e.g., Multi-Core DSP, FGPA, etc.) suffers from ineff…
OLLIE: Derivation-based Tensor Program Optimizer
Liyan Zheng, Haojie Wang, Jidong Zhai +7
Boosting the runtime performance of deep neural networks (DNNs) is critical due to their wide adoption in real-world tasks. Existing approaches to optimizing the tensor algebra exp…
ScalAna: Automating Scaling Loss Detection with Graph Analysis
Yuyang Jin, Haojie Wang, Teng Yu +4
Scaling a parallel program to modern supercomputers is challenging due to inter-process communication, Amdahl's law, and resource contention. Performance analysis tools for finding…