4 papers
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
Haoyu Zheng, Yongqiang Zhang, Fangcheng Fu +7
To schedule LLM inference, the \textit{shortest job first} (SJF) principle is favorable by prioritizing requests with short output lengths to avoid head-of-line (HOL) blocking. Exi…
Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
Haoyu Zheng, Fangcheng Fu, Jia Wu +6
LLM-based workflows compose specialized agents to execute complex tasks, and these agents usually share substantial context, allowing KV-Cache reuse to save computation. Existing a…
Generative Bid Shading in Real-Time Bidding Advertising
Yinqiu Huang, Hao Ma, Wenshuai Chen +7
Bid shading plays a crucial role in Real-Time Bidding (RTB) by adaptively adjusting the bid to avoid advertisers overspending. Existing mainstream two-stage methods, which first mo…
You Only Evaluate Once: A Tree-based Rerank Method at Meituan
Shuli Wang, Yinqiu Huang, Changhao Li +6
Reranking plays a crucial role in modern recommender systems by capturing the mutual influences within the list. Due to the inherent challenges of combinatorial search spaces, most…