310 citations · 607 across the 39 of their papers we have counts for
4 papers · 1 filter
Universal Model Routing for Efficient LLM Inference
Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat +9
Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt…
Cascade-Aware Training of Language Models
Congchao Wang, Sean Augenstein, Keith Rush +5
Reducing serving cost and latency is a fundamental concern for the deployment of language models (LMs) in business applications. To address this, cascades of LMs offer an effective…
Faster Cascades via Speculative Decoding
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat +4
Cascades and speculative decoding are two common approaches to improving language models' inference efficiency. Both approaches involve interleaving models of different sizes, but…
Language Model Cascades: Token-level uncertainty and beyond
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum +3
Recent advances in language models (LMs) have led to significant improvements in quality on complex NLP tasks, but at the expense of increased inference costs. Cascading offers a s…