You don't need long context

Peter Belcak, Saurav Muralidharan, Jindong Jiang +9

Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show acr…