Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference

Donghyeon Joo, Ramyad Hadidi, Soheil Feizi +1

The increasing size of large language models (LLMs) challenges their usage on resource-constrained platforms. For example, memory on modern GPUs is insufficient to hold LLMs that a…