Towards Low-bit Communication for Tensor Parallel LLM Inference

Harry Dong, Tyler Johnson, Minsik Cho +1

Tensor parallelism provides an effective way to increase server large language model (LLM) inference efficiency despite adding an additional communication cost. However, as server…