2 papers
cs.LG2026
Can an MLP Absorb Its Own Skip Connection?
Antonij Mijoski, Marko Karbevski
We study when a skip connection around a single-hidden-layer MLP can be absorbed into a residual-free MLP of the same width. We first show that for any architecture whose skip bran…
cs.LG2026
Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
Marko Karbevski, Antonij Mijoski
We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of…