7 papers
Trillion-Parameter MoE in a Box: Decoupling Memory Provisioning with High-Bandwidth Flash
Pengfei Xia, Tuo Hao, Shengwei Li +5
An MoE appliance for trillion-parameter models at low concurrency must host terabytes of weights on one node and serve prefill and decode with fixed resources. Combining operator a…
Enabling High-Bandwidth Flash for Generative Recommendation Serving with Write-Aware KV Cache Policy
Danni Peng, Kai Wu, Tianyu Zuo +2
Generative recommendation (GR) systems increasingly leverage user-level KV cache reuse to avoid recomputing long user histories. However, the growing KV cache capacity and bandwidt…
Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
Hui Zang, Pengfei Xia, Hong Liu +5
Mixture-of-Experts (MoE) architectures enable language models to achieve unprecedented scale via sparse activation. However, their inference performance is often limited by data mo…
FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location
Jiongjiong Gu, Jianfeng Wang, Zidong Han +19
Modern AI serving increasingly relies on NPUs for conventional inference and large language model serving. However, current NPU deployments commonly expose physical devices directl…
CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems
Ziyang Ma, Dingyi Zhang, Sichu Liang +4
Although large language model (LLM) based multi-agent systems (MAS) show their capability to solve complex tasks and achieve higher performance over single agent systems, they lead…
MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills
Yingyong Hou, Xinyuan Lao, Huimei Wang +10
Background: Agent skills are increasingly deployed as modular, reusable capability units in AI agent systems. Medical research agent skills require safeguards beyond general-purpos…
When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judges
Sichu Liang, Zhenglin Wang, Jiajia Chu +3
Multi-agent LLM systems routinely generate multiple candidate responses that are aggregated by an LLM judge. To reduce the dominant prefill cost in such pipelines, recent work advo…