Bi-directional Model Cascading with Proxy Confidence

David Warren, Mark Dras

Model Cascading, recently applied successfully to LLMs, is a simple but powerful technique that improves the efficiency of inference by selectively applying models of varying sizes…