7 citations · 9 across the 5 of their papers we have counts for
5 papers
Can an MLP Absorb Its Own Skip Connection Exactly?
Antonij Mijoski, Marko Karbevski
The benefits usually attributed to skip connections are optimization-theoretic: a smoother loss landscape and better gradient propagation. We ask a representational question instea…
Beyond Linearity in Attention Projections: The Case for Nonlinear Queries
Marko Karbevski
Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection may be set to identity without noticeable performance deterioration.…
Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
Marko Karbevski, Antonij Mijoski
We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of…
Ergodicity breaking in wealth dynamics: The case of reallocating geometric Brownian motion
Viktor Stojkoski, Marko Karbevski
A growing body of empirical evidence suggests that the dynamics of wealth within a population tends to be non-ergodic, even after rescaling the individual wealth with the populatio…
Evolution of cooperation in networked heterogeneous fluctuating environments
Viktor Stojkoski, Marko Karbevski, Zoran Utkovski +2
Fluctuating environments are situations where the spatio-temporal stochasticity plays a significant role in the evolutionary dynamics. The study of the evolution of cooperation in…