6 papers
Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving
Zhixin Wang, Zhengbo Wang, Fangcheng Fu +15
Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing…
MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
Yangjia Hu, Haodong Wang, Zicong Hong +8
4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs). However, its limited bit-width representation struggles…
EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management
Rongzhi Zhu, Xiang Huang, Yuchuan Wu +8
Emotional intelligence (EI) in Large Language Models (LLMs) is often evaluated through static understanding tasks or single-response dialogue generation. However, emotion managemen…
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
Weijie Shi, Qiang Xu, Fan Deng +9
Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with oppo…
Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimization
He Du, Qiming Ge, Jiakai Hu +18
We present Kernel-Smith, a framework for high-performance GPU kernel and operator generation that combines a stable evaluation-driven evolutionary agent with an evolution-oriented…
LocRef-Diffusion:Tuning-Free Layout and Appearance-Guided Generation
Fan Deng, Yaguang Wu, Xinyang Yu +4
Recently, text-to-image models based on diffusion have achieved remarkable success in generating high-quality images. However, the challenge of personalized, controllable generatio…