SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

Chuan Liu, Shuoming Zhang, Zhicheng Li +6

No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel…