6 citations · 6 across the 4 of their papers we have counts for
5 papers
CacheFlow: Efficient LLM Serving via Automated 3D-Parallel KV Cache Restoration
Sean Nian, Jiahao Fang, Hanzhang Shen +3
KV cache restoration is becoming a major bottleneck in long-context LLM serving, including multi-turn conversations, agentic workloads, and RAG. Existing advances largely treat res…
When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints
Yuheng Chen, Zhiyu Wu, Bowen Cheng +2
We identify and systematically characterize a class of task-structural alignment failures in large language models (LLMs): even when the harmful intent remains unchanged, changing…
The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization
Jae-Won Chung, Jeff J. Ma, Ruofan Wu +5
As the adoption of Generative AI in real-world services grow explosively, energy has emerged as a critical bottleneck resource. However, energy remains a metric that is often overl…
JITServe: SLO-aware LLM Serving with Imprecise Request Information
Wei Zhang, Zhiyu Wu, Yi Mu +5
The integration of Large Language Models (LLMs) into applications ranging from interactive chatbots to multi-agent systems has introduced a wide spectrum of service-level objective…
Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services
Jiachen Liu, Jae-Won Chung, Zhiyu Wu +3
Large language models (LLMs) are now at the core of conversational AI services such as real-time translation and chatbots, which provide live user interaction by incrementally stre…