4 papers
Monitoring latent reasoning in looped language models
Pawel Olszowiec, Paulina Tomaszewska, Maciej Żelaszczyk +2
Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising pop…
Same path, different walk: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG
Pawel Olszowiec, Michal Byra, Grzegorz Gruszczynski +2
Recurrent Transformers reusing their weights rather than stacking distinct layers are becoming widely adopted due to their parameter efficiency [1,2,3]. However, the exact repr…
Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision
Grzegorz Gruszczynski, Pawel Olszowiec, Michal Byra +2
Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-…
Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
Michal Byra, Pawel Olszowiec, Grzegorz Stefanski +2
Vision Transformers (ViTs) implement depth by stacking independently parameterized blocks, but it remains unclear how much of this parameterization is necessary and how much can be…