Compressing History into Memory: Distilling Transformers into Recurrent Transformers

Philippe Weinzaepfel, Christian Wolf, Bülent Mert Sariyildiz +3

Transformers are AI's workhorse but their computational cost becomes prohibitive when processing long sequences. We target long-horizon streaming vision and robotics applications,…