10 papers
EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
Kunming Shao, Jierun Chen, Jiangnan Yu +7
LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (…
PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention
Kunming Shao, Jierun Chen, Yanli Wang +4
At each decoding step a language model attends over the key-value (KV) cache of every earlier token, so at long context the attention call is bounded by memory bandwidth. Sparse at…
MoE Expert Execution in Disaggregated LLM Serving with a High-Bandwidth ReRAM Near-Memory Architecture
Kunming Shao, Ming Zeng, Xin Yuan +5
Attention-FFN disaggregation maps LLM modules to specialized pools, creating an opening to keep Mixture-of-Experts (MoE) weights resident in a high-bandwidth FFN pool. Decode SLOs,…
Balancing FP8 Computation Accuracy and Efficiency on Digital CIM via Shift-Aware On-the-fly Aligned-Mantissa Bitwidth Prediction
Liang Zhao, Kunming Shao, Zhipeng Liao +4
FP8 low-precision formats have gained significant adoption in Transformer inference and training. However, existing digital compute-in-memory (DCIM) architectures face challenges i…
DS-CIM: Digital Stochastic Computing-In-Memory Featuring Accurate OR-Accumulation via Sample Region Remapping for Edge AI Models
Kunming Shao, Liang Zhao, Jiangnan Yu +5
Stochastic computing (SC) offers hardware simplicity but suffers from low throughput, while high-throughput Digital Computing-in-Memory (DCIM) is bottlenecked by costly adder logic…
A Memory-Efficient Retrieval Architecture for RAG-Enabled Wearable Medical LLMs-Agents
Zhipeng Liao, Kunming Shao, Jiangnan Yu +5
With powerful and integrative large language models (LLMs), medical AI agents have demonstrated unique advantages in providing personalized medical consultations, continuous health…
DIRC-RAG: Accelerating Edge RAG with Robust High-Density and High-Loading-Bandwidth Digital In-ReRAM Computation
Kunming Shao, Zhipeng Liao, Jiangnan Yu +9
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge retrieval but faces challenges on edge devices due to high storage, ene…
A Flexible Precision Scaling Deep Neural Network Accelerator with Efficient Weight Combination
Liang Zhao, Kunming Shao, Fengshi Tian +3
Deploying mixed-precision neural networks on edge devices is friendly to hardware resources and power consumption. To support fully mixed-precision neural network inference, it is…
SynDCIM: A Performance-Aware Digital Computing-in-Memory Compiler with Multi-Spec-Oriented Subcircuit Synthesis
Kunming Shao, Fengshi Tian, Xiaomeng Wang +12
Digital Computing-in-Memory (DCIM) is an innovative technology that integrates multiply-accumulation (MAC) logic directly into memory arrays to enhance the performance of modern AI…
Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning
Xiaoyu Gan, Jingbo Jiang, Jingyang Zhu +3
Substantial efforts have been devoted to alleviating the impact of the long-tailed class distribution in federated learning. In this work, we observe an interesting phenomenon that…