Latency-Aware Adaptive Best-of-$N$

Alon Jacobson, Qian Xie, Peter I. Frazier

Test-time compute (TTC) improves language model responses by spending additional computation at inference time, and the right amount of compute depends on the prompt. Adaptive TTC…