4 papers · 1 filter
End-to-End Context Compression at Scale
Ang Li, Sean McLeish, Haozhe Chen +12
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degra…
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
Colin White, Samuel Dooley, Manley Roberts +15
Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render ben…
Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
Ryan Synk, Monte Hoover, John Kirchenbauer +6
There is growing demand for performing inference with hundreds of thousands of input tokens on trained transformer models. Inference at this extreme scale demands significant compu…
GenQA: Generating Millions of Instructions from a Handful of Prompts
Jiuhai Chen, Rifaa Qadri, Yuxin Wen +4
Most public instruction finetuning datasets are relatively small compared to the closed source datasets used to train industry models. To study questions about finetuning at scale,…