1 citations · 2 across the 12 of their papers we have counts for
4 papers · 1 filter
Transformer Heads Looking for Order
Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser +4
In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transforme…
Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models
Inez Okulska, Bartosz Naskręcki, Jan Piotrowski +1
Prompt echoing is a recognized failure mode of instruct language models, in which a model instead of generating a response, mirrors the provided prompt, even though it did not rece…
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
Tomasz Steifer
We investigate the expressive power of hybrid recurrent-attention decoders, a class of architectures used in recent open-source language models such as Qwen3-Next and its successor…
Parity, Sensitivity, and Transformers
Alexander Kozachinskiy, Tomasz Steifer, Przemysław Wałȩga
Understanding what neural architectures can and cannot compute is a central challenge in the theory of AI. One of the fundamental problems in this context is the PARITY task, which…