7 papers · 1 filter
Short-Context Dominance: How Much Local Context Natural Language Actually Needs?
Vala Vakilian, Zimeng Wang, Ankit Singh Rawat +1
We investigate the short-context dominance hypothesis: that for most sequences, a small local prefix suffices to predict their next tokens. Using large language models as statistic…
Universal Model Routing for Efficient LLM Inference
Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat +9
Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt…
Faster Cascades via Speculative Decoding
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat +4
Cascades and speculative decoding are two common approaches to improving language models' inference efficiency. Both approaches involve interleaving models of different sizes, but…
Analysis of Plan-based Retrieval for Grounded Text Generation
Ameya Godbole, Nicholas Monath, Seungyeon Kim +3
In text generation, hallucinations refer to the generation of seemingly coherent text that contradicts established knowledge. One compelling hypothesis is that hallucinations occur…
Cascade-Aware Training of Language Models
Congchao Wang, Sean Augenstein, Keith Rush +5
Reducing serving cost and latency is a fundamental concern for the deployment of language models (LMs) in business applications. To address this, cascades of LMs offer an effective…
Think before you speak: Training Language Models With Pause Tokens
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat +3
Language models generate responses by producing a series of tokens in immediate succession: the token is an outcome of manipulating hidden vectors per layer, one v…