4 papers
Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
Kangqi Ni, Wenyue Hua, Xiaoxiang Shi +3
Multi-agent applications often execute complex tasks as multi-stage workflows, where each stage is an LLM call whose output becomes part of context for subsequent steps. Existing L…
Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees
Shi Chang, Boyuan Chen, Kishanthan Thangarajah +2
Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows, yet efficiently serving them in resource-constrained, self-hosted envi…
Context-Aware CodeLLM Eviction for AI-assisted Coding
Kishanthan Thangarajah, Boyuan Chen, Shi Chang +1
AI-assisted coding tools powered by Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows. To address concerns around privacy…
Software Performance Engineering for Foundation Model-Powered Software
Haoxiang Zhang, Shi Chang, Arthur Leung +4
The rise of Foundation Models (FMs) like Large Language Models (LLMs) is revolutionizing software development. Despite the impressive prototypes, transforming FMware into productio…