Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling

Luping Wang, Weigao Chen, Yifei Wu +11

Model-as-a-service platforms call for continuous optimization as complex serving conditions expose inefficiencies missed before deployment. Detailed always-on profiling can incur s…