Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models

Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh +5

Deploying large language models (LLMs) in real-world applications is often hindered by strict computational and latency constraints. While dynamic inference offers the flexibility…