11 citations · 12 across the 4 of their papers we have counts for
6 papers
EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
Kunming Shao, Jierun Chen, Jiangnan Yu +7
LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (…
PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention
Kunming Shao, Jierun Chen, Yanli Wang +4
At each decoding step a language model attends over the key-value (KV) cache of every earlier token, so at long context the attention call is bounded by memory bandwidth. Sparse at…
31.1 A 14.08-to-135.69Token/s ReRAM-on-Logic Stacked Outlier-Free Large-Language-Model Accelerator with Block-Clustered Weight-Compression and Adaptive Parallel-Speculative-Decoding
Pingcheng Dong, Yonghao Tan, Xuejiao Liu +13
This work presents a 55nm speculative decoding-based LLM accelerator with bumping-based face-to-face ReRAM-on-logic stacking technology. It features a local rotation unit for outli…
A 28nm 0.22μJ/token memory-compute-intensity-aware CNN-Transformer accelerator with hybrid-attention-based layer-fusion and cascaded pruning for semantic-segmentation
Pingcheng Dong, Yonghao Tan, Xuejiao Liu +14
This work presents a 28nm 13.93mm2 CNN-Transformer accelerator for semantic segmentation, achieving 3.86-to-10.91x energy reduction over previous designs. It features a hybrid atte…
A Memory-Efficient Retrieval Architecture for RAG-Enabled Wearable Medical LLMs-Agents
Zhipeng Liao, Kunming Shao, Jiangnan Yu +5
With powerful and integrative large language models (LLMs), medical AI agents have demonstrated unique advantages in providing personalized medical consultations, continuous health…
A 137.5 TOPS/W SRAM Compute-in-Memory Macro with 9-b Memory Cell-Embedded ADCs and Signal Margin Enhancement Techniques for AI Edge Applications
Xiaomeng Wang, Fengshi Tian, Xizi Chen +7
In this paper, we propose a high-precision SRAM-based CIM macro that can perform 4x4-bit MAC operations and yield 9-bit signed output. The inherent discharge branches of SRAM cells…