What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute

Zhixu Du, Weijia Han, Hai Helen Li +1

Large language models spend the same amount of computation on every token they generate, regardless of how difficult each token is to produce. Methods such as speculative decoding…