Large Language Model Partitioning for Low-Latency Inference at the Edge

Dimitrios Kafetzis, Ramin Khalili, Iordanis Koutsopoulos

Large Language Models (LLMs) based on autoregressive, decoder-only Transformers generate text one token at a time, where a token represents a discrete unit of text. As each newly p…