3 papers
cs.DC2025
Memory Offloading for Large Language Model Inference with Latency SLO Guarantees
Chenxiang Ma, Zhisheng Ye, Hanyu Zhao +9
Offloading large language models (LLMs) state to host memory during inference promises to reduce operational costs by supporting larger models, longer inputs, and larger batch size…
cs.AR2024
Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM
Zhongkai Yu, Shengwen Liang, Tianyun Ma +12
Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience w…
cs.AR2024
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Xiurui Pan, Endian Li, Qiao Li +6
The widespread of Large Language Models (LLMs) marks a significant milestone in generative AI. Nevertheless, the increasing context length and batch size in offline LLM inference e…