5 papers · 1 filter
When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
Przemyslaw Forys, Haoran Wu, Can Xiao +8
Agentic inference now dominates the LLM inference landscape, requiring LLMs to actively engage in multi-turn interactions with tool-calling capabilities. This introduces a more com…
Open Problems in Constitutional Preference Reconstruction
Eleanor Clifford, Michael Amir, Arduin Findeis +2
Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a \emph{choice}, not the rationale behind it. Methods s…
MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUs
Haoran Wu, Zeyu Cao, Yao Lai +16
Emerging agentic large language model (LLM) workloads are driving rapidly growing demand for memory capacity and bandwidth. Different phases of inference, such as prefill and decod…
Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference
Yiren Zhao, Junyi Liu
AI agent inference is driving an inference heavy datacenter future and exposes bottlenecks beyond compute - especially memory capacity, memory bandwidth and high-speed interconnect…
NPU Design for Diffusion Language Model Inference
Binglei Lou, Haoran Wu, Kevin Lau +9
Diffusion-based LLMs (dLLMs) fundamentally depart from traditional autoregressive (AR) LLM inference: they leverage bidirectional attention, block-wise KV cache refreshing, cross-s…