HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving

Avinash Kumar, Shashank Nag, Jason Clemons +2

Early-Exit Large Language Models (EE-LLMs) enable high throughput inference by allowing tokens to exit early at intermediate layers. However, their throughput is limited by the com…