Dynamic Rebatching for Efficient Early-Exit Inference with DREX

Xuting Liu, Daniel Alexander, Siva Kesava Reddy Kakarla +2

Early-Exit (EE) is a Large Language Model (LLM) architecture that accelerates inference by allowing easier tokens to be generated using only a subset of the model's layers. However…