7 papers
NORmal: Architecting 3D Ferroelectric NOR Flash as Main Memory -- From Co-Fabricated BEOL Arrays to 200B-Class Multi-Agent LLMs in a Smartphone
Zhuoran Li, Chen Gu, Zhuohang Bian +9
Smartphone agents increasingly work as teams, in which a large planner directs parallel workers across long sessions. Running such a team on the device requires 200B-class models a…
HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving
Zhuoran Li, Zhuohang Bian, Xin Huang +3
A faster storage device should make serving faster. We find the opposite. High-Bandwidth Flash (HBF) stacks NAND behind a wide, package-local interface, promising flash-scale capac…
TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference
Zhuoran Li, Zhuohang Bian, Zihao Huang +4
Large language model (LLM) serving is now limited by the key-value (KV) cache. During decode, each new token rereads prior KV state, so attention becomes a bandwidth- and capacity-…
TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
Zhuohang Bian, Feiyang Wu, Chengrui Zhang +3
Multi-agent LLM applications organize execution in synchronized rounds where a central scheduler gathers outputs from all agents and redistributes the combined context. This All-Ga…
Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
Simin Li, Zihao Mao, Hanxiao Li +13
In cooperative Multi-Agent Reinforcement Learning (MARL), it is a common practice to tune hyperparameters in ideal simulated environments to maximize cooperative performance. Howev…
TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
Zhuohang Bian, Feiyang Wu, Zhuoran Li +2
Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that rely on external function calls. This workload creates severe performance challenges…
TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
Feiyang Wu, Zhuohang Bian, Guoyang Duan +6
The increasing demand for large language model (LLM) serving has necessitated significant advancements in the optimization and profiling of LLM inference systems. As these models b…