Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator

Cong Li, Yihan Yin, Chenhao Xue +7

Large language models (LLMs) have been widely deployed for online generative services, where numerous LLM instances jointly handle workloads with fluctuating request arrival rates…