4 papers · 1 filter
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
Cheuk Hang Lau, Zeyu Cao, Kevin Wong Cheuk Yin +6
The optimization of LLM serving engines, such as vLLM and SGLang, is largely benchmark-driven: optimizations, scheduling policies, hardware and system designs are all selected base…
DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs
Zeyu Cao, Xuan Guo, Cheng Zhang +3
As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a produ…
Reasoning Compression with Mixed-Policy Distillation
Han Yang, Mingyan Wu, Bailan He +4
Reasoning-centric large language models (LLMs) achieve strong performance by generating intermediate reasoning trajectories, but often incur excessive token usage and high inferenc…
KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware
Jiayi Nie, Haoran Wu, Yao Lai +9
New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels, a time-consuming and error-prone process that does…