Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

Liming Liu, Mingze Wang, Tuo Zhao

As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training. In typical serving, prompt prefill runs in parall…