From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades

Yu Wu, Shuo Wu, Ye Tao +2

Standard LLM cascades improve efficiency by deferring difficult queries from weak to strong models. However, these systems are typically static: when faced with repeated or semanti…