ISO: Overlap of Computation and Communication within Seqenence For LLM Inference

Bin Xiao, Lei Su

In the realm of Large Language Model (LLM) inference, the inherent structure of transformer models coupled with the multi-GPU tensor parallelism strategy leads to a sequential exec…