3 papers
cs.CL2025
Integral Transformer: Denoising Attention, Not Too Much Not Too Little
Ivan Kobyzev, Abbas Ghaddar, Dingtao Hu +1
Softmax self-attention often assigns disproportionate weight to semantically uninformative tokens such as special tokens and punctuation, a phenomenon known as attention noise. Whi…
cs.CL2025
ReGLA: Refining Gated Linear Attention
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh +2
Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also…
cs.LG2025
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
Peng Lu, Ahmad Rashid, Ivan Kobyzev +2
Regularization techniques are crucial to improving the generalization performance and training efficiency of deep neural networks. Many deep learning algorithms rely on weight deca…