AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism

Wendong Xu, Chujie Chen, He Xiao +8

Large Language Model (LLM) inference services demand exceptionally high availability and low latency, yet multi-GPU Tensor Parallelism (TP) makes them vulnerable to single-GPU fail…