From the 1 of 15 linked papers with an AI index.
6 papers · 1 filter
Feed-Forward Steering in Transformer Residual Dynamics
Timur Mudarisov, Mikhail Burtsev, Radu State
Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere. We extend this framework by incorporating the feed-forward network (FF…
Geometry-Guided Layerwise FFN Width Allocation in Transformers
Timur Mudarisov, Mikhail Burtsev, Radu State
Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth. We ask whether this capacity can inste…
Limitations of Normalization in Attention Mechanism
Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova +1
This paper investigates the limitations of the normalization in attention mechanisms. We begin with a theoretical framework that enables the identification of the model's selective…
Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling
Ivan Rodkin, Daniil Orel, Konstantin Smirnov +9
Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear. We study this question in a controlled cellular-autom…
Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Training
Artyom Sorokin, Nazar Buzun, Alexander Anokhin +7
Retrieval-Augmented Generation (RAG) methods enhance LLM performance by efficiently filtering relevant context for LLMs, reducing hallucinations and inference cost. However, most e…
SRMT: Shared Memory for Multi-agent Lifelong Pathfinding
Alsu Sagirova, Yuri Kuratov, Mikhail Burtsev
Coordination in decentralized multi-agent reinforcement learning (MARL) necessitates that agents share information about their behavior and intentions. Existing approaches rely on…