6 papers · 1 filter
Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
Xunzhuo Liu, Bowei He, Xue Liu +3
Production vLLM fleets typically provision each instance for the worst-case context length, leading to substantial KV-cache over-allocation and under-utilized concurrency. In pract…
Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents
Xunzhuo Liu, Bowei He, Xue Liu +3
Production AI agents frequently receive user-specific queries that are highly repetitive, with up to 47\% being semantically similar to prior interactions, yet each query is typica…
Adaptive Vision-Language Model Routing for Computer Use Agents
Xunzhuo Liu, Bowei He, Xue Liu +3
Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Langua…
98 Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router
Xunzhuo Liu, Bowei He, Xue Liu +3
System-level routers that intercept LLM requests for safety classification, domain routing, and PII detection must be both fast and operationally lightweight: they should add minim…
Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
Alberto Purpura, Sahil Wadhwa, Jesse Zymet +5
The rapid growth of Large Language Models (LLMs) presents significant privacy, security, and ethical concerns. While much research has proposed methods for defending LLM systems ag…
Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment
Melissa Kazemi Rad, Huy Nghiem, Andy Luo +3
Large Language Models (LLMs) have demonstrated powerful capabilities that render them valuable in different applications, including conversational AI products. It is paramount to e…