Thinking Costs Tokens: When More Structure is Worth the Price

Thomas Nolasque, John Grey, Calista Pham +1

Adding inference structure to a language model lets it search, verify, and revise, but these actions consume the very budget they are supposed to use well. In this paper, we invest…