7 citations · 9 across the 5 of their papers we have counts for
Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Can an MLP Absorb Its Own Skip Connection Exactly?
Antonij Mijoski, Marko Karbevski
The benefits usually attributed to skip connections are optimization-theoretic: a smoother loss landscape and better gradient propagation. We ask a representational question instea…
cs.LG2026
Beyond Linearity in Attention Projections: The Case for Nonlinear Queries
Marko Karbevski
Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection may be set to identity without noticeable performance deterioration.…
cs.LG2025
Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
Marko Karbevski, Antonij Mijoski
We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of…