3 citations · 4 across the 3 of their papers we have counts for
3 papers
Linear Self-Attention Approximation via Trainable Feedforward Kernel
Uladzislau Yorsh, Alexander Kovalenko
In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches -- models attaining sub-quadratic attention complexity can utilize a notion…
SimpleTRON: Simple Transformer with O(N) Complexity
Uladzislau Yorsh, Alexander Kovalenko, Vojtěch Vančura +3
In this paper, we propose that the dot product pairwise matching attention layer, which is widely used in Transformer-based models, is redundant for the model performance. Attentio…
Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks
Alexander Kovalenko, Pavel Kordík, Magda Friedjungová
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks, where now excessively large models are…