Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions

Blanka Köver, Alexandra Butoi, Anej Svete +2

Transformers consistently fail to learn certain simple functions that are provably expressible with specific parameter settings. This gap between learnability and expressivity is p…