Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers

Peilong Zhou, Hongtao Zhang, Yulong Huang +3

Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known to overthink: iterating past…