CoDec: Prefix-Shared Decoding Kernel for LLMs

Zhibin Wang, Rui Ning, Chao Fang +12

Prefix-sharing among multiple prompts presents opportunities to combine the operations of the shared prefix, while attention computation in the decode stage, which becomes a critic…