2 papers
cs.CL2024
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
Bin Gao, Zhuomin He, Puru Sharma +6
Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conver…
cs.OS2024
numaPTE: Managing Page-Tables and TLBs on NUMA Systems
Bin Gao, Qingxuan Kang, Hao-Wei Tee +3
Memory management operations that modify page-tables, typically performed during memory allocation/deallocation, are infamous for their poor performance in highly threaded applicat…