Understanding Dynamic Compute Allocation in Recurrent Transformers

Ibraheem Muhammad Moosa, Suhas Lohit, Ye Wang +2

Token-level adaptive computation seeks to reduce inference cost by allocating more computation to harder tokens and less to easier ones. However, prior work is primarily evaluated…