Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding

Arun Ramachandran, Ramaswamy Govindarajan, Murali Annavaram +4

With the skyrocketing costs of GPUs and their virtual instances in the cloud, there is a significant desire to use CPUs for large language model (LLM) inference. KV cache update, o…