vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving

Jiale Xu, Rui Zhang, Cong Guo +11

Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. This surge in demand poses significant challenges in optimizing throughp…