Write Back the $\Delta$: Revisiting the Same Tokens with Fresh Representations

Wencheng Ye, Anning Hu, Xiangdong Zhang +5

Transformers process information strictly forward through depth, preventing deeper computation from revisiting and refining earlier representations. To augment the standard forward…