SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference

Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal +3

Autoregressive large language models (LLMs) have made remarkable progress in various natural language generation tasks. However, they incur high computation cost and latency result…