4 papers
MemExplorer: Navigating the Heterogeneous Memory Design Space for Agentic Inference NPUs
Haoran Wu, Zeyu Cao, Yao Lai +15
Emerging agentic LLM workloads are driving rapidly growing demand on both memory capacity and bandwidth, with different phases of inference (e.g., prefill and decode) imposing dist…
NPU Design for Diffusion Language Model Inference
Binglei Lou, Haoran Wu, Kevin Lau +9
Diffusion-based LLMs (dLLMs) fundamentally depart from traditional autoregressive (AR) LLM inference: they leverage bidirectional attention, block-wise KV cache refreshing, cross-s…
Group Policy Gradient
Junhua Chen, Zixi Zhang, Hantao Zhong +1
We introduce Group Policy Gradient (GPG), a family of critic-free policy-gradient estimators for general MDPs. Inspired by the success of GRPO's approach in Reinforcement Learning…
Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
Haoran Wu, Can Xiao, Jiayi Nie +15
LLMs now form the backbone of AI agents across a diverse range of applications, including tool use, command-line interfaces, and web or computer interaction. These agentic LLM infe…